Oct 30 06:53:57.599397 ip-10-0-139-124 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 06:53:57.599408 ip-10-0-139-124 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 06:53:57.599416 ip-10-0-139-124 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 06:53:57.599638 ip-10-0-139-124 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 06:54:07.848622 ip-10-0-139-124 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 06:54:07.848637 ip-10-0-139-124 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot eb5d9a8daf65478b8ff08cc75654e403 --
Oct 30 06:56:06.502600 ip-10-0-139-124 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 06:56:06.902654 ip-10-0-139-124 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:06.902654 ip-10-0-139-124 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 06:56:06.902654 ip-10-0-139-124 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:06.902654 ip-10-0-139-124 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:06.907462 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907376    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.907462 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907460    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.907462 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907465    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.907462 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907468    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907471    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907474    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907477    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907480    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907482    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907485    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907487    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907495    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907500    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907502    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907505    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907508    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907511    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907513    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907516    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907519    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907521    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907523    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.907583 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907526    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907529    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907531    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907534    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907537    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907539    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907543    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907545    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907548    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907551    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907553    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907556    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907559    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907562    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907564    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907568    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907570    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907573    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907575    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907578    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.908021 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907580    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907590    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907594    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907596    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907599    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907602    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907605    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907608    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907611    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907613    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907616    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907618    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907621    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907623    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907626    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907628    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907631    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907633    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907655    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.908540 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907659    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907662    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907668    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907673    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907676    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907679    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907682    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907684    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907687    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907689    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907692    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907694    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907697    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907699    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907702    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907710    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907713    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907716    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907719    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.908979 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907721    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907724    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907727    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907729    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907731    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907734    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907736    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907739    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907741    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907744    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907748    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907752    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907754    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907757    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907760    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907765    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907769    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907775    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907778    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.909444 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907781    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907783    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907786    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907789    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907791    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.909878 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.907796    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.910174 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910152    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.910174 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910174    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910179    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910222    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910233    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910237    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910241    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910247    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910251    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910255    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910259    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910263    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910267    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910272    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910276    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.910273 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910281    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910303    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910308    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910313    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910317    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910321    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910326    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910330    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910334    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910337    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910342    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910346    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910350    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910354    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910363    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910366    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910371    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910375    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910379    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910383    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.910882 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910387    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910417    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910422    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910426    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910430    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910434    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910438    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910447    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910451    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910455    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910459    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910463    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910467    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910471    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910477    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910481    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910485    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910490    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910495    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910499    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.911535 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910508    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910512    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910516    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910521    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910525    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910529    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910534    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910538    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910542    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910546    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910550    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910554    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910563    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910567    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910596    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910605    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910611    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910617    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910621    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.912131 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910626    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910630    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910634    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910639    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910643    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910653    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910657    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910662    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910668    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910672    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910677    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910681    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910686    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910690    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910695    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910699    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910705    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910709    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910718    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.912645 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910722    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910726    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910733    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910739    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910743    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910748    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910752    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910759    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910763    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910769    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910778    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910782    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.910786    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912085    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912097    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912101    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912104    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912107    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912110    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.913100 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912113    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912116    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912119    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912121    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912124    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912127    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912131    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912135    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912139    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912142    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912145    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912148    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912151    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912153    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912156    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912159    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912161    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912164    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912166    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.913570 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912169    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912171    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912174    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912177    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912180    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912182    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912185    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912188    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912191    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912194    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912196    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912199    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912201    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912204    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912206    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912209    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912212    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912214    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912217    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912219    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.914096 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912223    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912226    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912228    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912231    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912233    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912236    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912239    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912241    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912244    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912246    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912249    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912252    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912254    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912257    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912259    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912262    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912264    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912267    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.914586 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912273    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912276    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912278    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912281    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912283    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912286    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912289    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912305    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912308    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912311    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912313    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912316    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912321    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912324    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912327    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912329    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912332    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912334    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912337    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.915032 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912340    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912342    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912345    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912348    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912350    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912353    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912356    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912360    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912363    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912366    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912370    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912372    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912375    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912378    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912381    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912384    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912387    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912389    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912391    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.915523 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912394    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912396    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.912399    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913057    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913066    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913073    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913077    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913082    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913086    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913091    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913095    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913099    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913102    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913106    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913109    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913112    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913116    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913119    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913122    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913125    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913128    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913133    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913136    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 06:56:06.915970 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913140    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913143    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913146    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913151    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913154    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913157    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913161    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913164    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913167    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913170    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913173    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913177    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913181    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913184    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913187    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913191    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913195    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913198    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913202    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913205    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913208    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913211    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913215    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913220    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 06:56:06.916530 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913223    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913226    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913229    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913233    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913236    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913239    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913242    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913245    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913248    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913251    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913255    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913259    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913262    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913265    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913269    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913272    2574 flags.go:64] FLAG: --help="false"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913275    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-139-124.ec2.internal"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913278    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913281    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913284    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913288    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913304    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913308    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 06:56:06.917137 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913311    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913315    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913318    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913321    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913324    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913327    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913330    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913333    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913336    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913339    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913342    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913345    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913348    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913351    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913357    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913360    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913363    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913366    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913369    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913372    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913376    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913379    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913383    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913386    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913390    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 06:56:06.917745 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913394    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913397    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913403    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913406    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913409    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913412    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913415    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913423    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913426    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913430    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913433    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913436    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913440    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913443    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913446    2574 flags.go:64] FLAG: --port="10250"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913449    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913452    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-004c015cc3e17b04a"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913455    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913458    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913461    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913466    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913470    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913473    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913476    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913479    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913483    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 06:56:06.918349 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913486    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913490    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913493    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913496    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913499    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913502    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913505    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913508    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913512    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913515    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913518    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913522    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913525    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913528    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913531    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913534    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913537    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913541    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913544    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913549    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913552    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913555    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913560    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913563    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913566    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 06:56:06.918966 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913568    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913573    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913576    2574 flags.go:64] FLAG: --v="2"
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913581    2574 flags.go:64] FLAG: --version="false"
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913586    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913590    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.913594    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913747    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913751    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913755    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913757    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913760    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913763    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913765    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913768    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913772    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913775    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913777    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913780    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913782    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913785    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.919558 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913788    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913790    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913794    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913797    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913800    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913802    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913805    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913808    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913810    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913813    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913815    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913818    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913822    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913824    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913827    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913829    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913832    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913835    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913837    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913840    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.920107 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913842    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913845    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913847    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913850    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913853    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913855    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913860    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913862    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913865    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913868    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913870    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913873    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913875    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913878    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913881    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913883    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913886    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913889    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913892    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913894    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.920601 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913897    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913900    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913902    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913905    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913909    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913912    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913914    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913917    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913920    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913922    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913925    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913928    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913930    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913933    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913935    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913938    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913940    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913943    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913947    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.921074 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913950    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913953    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913956    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913959    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913961    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913964    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913966    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913969    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913972    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913974    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913977    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913980    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913985    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913988    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913992    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913995    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.913998    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914002    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914006    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.921572 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914009    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914012    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914015    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914017    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914020    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914023    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914025    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914028    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914031    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914033    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914037    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914039    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914043    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914046    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.914072    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914113    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.922015 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914118    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914121    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914124    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914126    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914129    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914132    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914134    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914137    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914140    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914142    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914145    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914147    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914150    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914153    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914157    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914159    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914162    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914164    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914167    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914169    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.922408 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914172    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914174    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914177    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914180    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914182    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914185    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914187    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914190    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914192    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914195    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914198    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914200    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914203    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914205    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914208    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914210    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914213    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914215    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914218    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914220    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.922872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914223    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914226    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914229    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914231    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914234    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914236    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914240    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914242    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914245    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914247    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914250    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914252    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914254    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914257    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914260    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914262    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914265    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914267    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914270    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.923368 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914273    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914275    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914278    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914280    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914283    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914286    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914289    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914306    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914310    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914314    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914317    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914320    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914322    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914326    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914330    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914333    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914337    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914340    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914343    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.923821 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914349    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914351    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914354    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914357    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914361    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914364    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914367    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914370    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914373    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914375    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914378    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914381    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914384    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914387    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914389    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914392    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914394    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914397    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914399    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914402    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.924271 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914404    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914407    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914409    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914413    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914416    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914418    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.914420    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.922470    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.922487    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.922501    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.922505    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922566    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922573    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922577    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922581    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922584    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922587    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922590    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922592    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.924755 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922595    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922597    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922600    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922603    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922606    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922608    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922611    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922614    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922617    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922619    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922621    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922624    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922627    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922629    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922632    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922634    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922637    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922639    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922641    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922644    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.925187 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922647    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922651    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922654    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922657    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922659    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922662    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922664    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922668    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922670    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922673    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922676    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922678    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922681    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922684    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922686    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922689    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922692    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922694    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922697    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.925823 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922699    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922703    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922707    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922710    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922713    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922716    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922719    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922722    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922724    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922727    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922729    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922732    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922735    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922737    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922741    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922743    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922747    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922750    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922752    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922755    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.926259 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922758    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922760    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922763    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922766    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922768    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922771    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922773    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922776    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922779    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922781    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922784    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922786    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922789    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922791    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922797    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922799    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922802    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922805    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922808    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.926735 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922810    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922813    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922815    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922818    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922821    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922823    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922827    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922829    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922832    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922835    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922837    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922840    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922842    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922845    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922848    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922850    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922853    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922855    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.927192 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.922858    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.922886    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923039    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923043    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923046    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923049    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923051    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923054    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923056    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923059    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923062    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923064    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923067    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923070    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923073    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923075    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.927633 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923078    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923080    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923083    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923085    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923088    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923091    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923093    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923096    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923099    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923102    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923105    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923107    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923109    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923112    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923115    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923118    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923120    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923122    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923125    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.928007 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923129    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923134    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923138    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923141    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923143    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923146    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923148    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923151    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923154    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923156    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923159    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923162    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923164    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923166    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923169    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923172    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923174    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923177    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923180    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923182    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.928477 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923185    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923188    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923191    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923194    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923196    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923199    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923201    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923204    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923206    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923209    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923211    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923214    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923216    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923220    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923222    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923225    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923227    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923230    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923232    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923235    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.928971 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923237    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923240    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923242    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923245    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923247    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923250    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923252    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923255    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923257    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923260    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923263    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923265    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923268    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923270    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923273    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923276    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923278    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923281    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923283    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923286    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.929510 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923288    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923305    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923308    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923311    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923314    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923316    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923323    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923327    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923330    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923332    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923335    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923338    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.923361    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923399    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:06.929974 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923403    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923406    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923409    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923411    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923414    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923416    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923420    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923423    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923426    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923428    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923431    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923434    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923436    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923445    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923448    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923450    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923453    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923455    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923458    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923460    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:06.930374 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923463    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923465    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923468    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923470    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923473    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923475    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923478    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923480    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923482    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923485    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923488    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923490    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923492    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923495    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923497    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923500    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923502    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923505    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923507    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923510    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:06.930883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923513    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923515    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923517    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923520    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923522    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923525    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923528    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923530    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923533    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923535    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923537    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923540    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923544    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923548    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923551    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923554    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923558    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923560    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923563    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:06.931365 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923565    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923568    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923570    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923573    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923578    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923580    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923583    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923586    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923588    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923591    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923594    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923596    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923599    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923602    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923604    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923607    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923609    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923612    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923614    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923617    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:06.931813 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923620    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923622    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923625    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923628    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923630    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923633    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923635    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923638    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923640    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923643    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923645    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923648    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923652    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923656    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923659    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923662    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923666    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923668    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923671    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:06.932357 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923675    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923677    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923680    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923682    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923685    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923687    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:06.923690    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.923747    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.926040    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.926930    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.927028    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.927064    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.928786    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:56:06.932815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.930018    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:56:06.938044 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.938022    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 06:56:06.941516 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.941500    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 06:56:06.942358 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.942335    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 06:56:06.946905 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.946885    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81221195-5e9f-4a8d-af0a-8aa778b15f34:/dev/nvme0n1p4 a5bc53f9-abac-4222-81c4-0bf3cb698942:/dev/nvme0n1p3]
Oct 30 06:56:06.946967 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.946905    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 06:56:06.953777 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.953656    2574 manager.go:217] Machine: {Timestamp:2026-10-30 06:56:06.952381695 +0000 UTC m=+0.338024367 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098442 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b44a83e7fb9c4814e32cb1c3d1920 SystemUUID:ec2b44a8-3e7f-b9c4-814e-32cb1c3d1920 BootID:eb5d9a8d-af65-478b-8ff0-8cc75654e403 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:12:00:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:12:00:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:c3:a4:35:17:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 06:56:06.953777 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.953765    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 06:56:06.953925 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.953850    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 06:56:06.957305 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957264    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 06:56:06.957438 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957306    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 06:56:06.957503 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957440    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 06:56:06.957503 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957447    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 06:56:06.957503 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957488    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:56:06.957618 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.957507    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:56:06.958636 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.958625    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 06:56:06.963094 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963080    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 06:56:06.963232 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963220    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 06:56:06.963729 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963718    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 06:56:06.963782 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963732    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 06:56:06.963782 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963742    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 06:56:06.963782 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963751    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 06:56:06.963782 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.963759    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 06:56:06.964166 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.964143    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:56:06.966248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.966226    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 06:56:06.973154 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.973138    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:56:06.974007 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.973992    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 06:56:06.974049 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974030    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 06:56:06.974234 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974223    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 06:56:06.974263 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974241    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 06:56:06.974263 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974251    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 06:56:06.974263 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974259    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 06:56:06.974358 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974269    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 06:56:06.974358 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974277    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 06:56:06.974769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974759    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 06:56:06.974801 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974772    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 06:56:06.974801 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974783    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 06:56:06.974801 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974791    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 06:56:06.974871 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974809    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 06:56:06.974871 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.974821    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 06:56:06.975286 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.975277    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 06:56:06.975331 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.975303    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 06:56:06.979793 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.979770    2574 server.go:1263] "Started kubelet"
Oct 30 06:56:06.979896 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.979840    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 06:56:06.980021 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.979947    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 06:56:06.980082 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.980068    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 06:56:06.980735 ip-10-0-139-124 systemd[1]: Started Kubernetes Kubelet.
Oct 30 06:56:06.981769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.981570    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 06:56:06.982752 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.982734    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 06:56:06.985374 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:06.985343    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 06:56:06.987918 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.987898    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 06:56:06.988629 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.988612    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 06:56:06.988728 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:06.988708    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:06.989136 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989106    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 06:56:06.989136 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989108    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 06:56:06.989136 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989138    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 06:56:06.989329 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989248    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 06:56:06.989329 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989256    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 06:56:06.989652 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989633    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 06:56:06.989652 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989646    2574 factory.go:55] Registering systemd factory
Oct 30 06:56:06.989652 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989653    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 06:56:06.989944 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989930    2574 factory.go:153] Registering CRI-O factory
Oct 30 06:56:06.989990 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989947    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 06:56:06.989990 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.989964    2574 factory.go:103] Registering Raw factory
Oct 30 06:56:06.990051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.990023    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 06:56:06.991094 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:06.991080    2574 manager.go:319] Starting recovery of all containers
Oct 30 06:56:07.005570 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.005551    2574 manager.go:324] Recovery completed
Oct 30 06:56:07.009697 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.009680    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.010043 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.010021    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 06:56:07.010122 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.010019    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 06:56:07.010991 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.010021    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-124.ec2.internal.18e33c4a6d2148d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-124.ec2.internal,},FirstTimestamp:2026-10-30 06:56:06.979733718 +0000 UTC m=+0.365376393,LastTimestamp:2026-10-30 06:56:06.979733718 +0000 UTC m=+0.365376393,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-124.ec2.internal,}"
Oct 30 06:56:07.011576 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.011561    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.011636 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.011589    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.011636 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.011599    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.012306 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012276    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.012408 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012277    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 06:56:07.012408 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012380    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 06:56:07.012408 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012400    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 06:56:07.012952 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012935    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.013015 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012965    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.013015 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.012976    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.014215 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.014202    2574 policy_none.go:50] "Start"
Oct 30 06:56:07.014284 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.014219    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 06:56:07.014284 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.014232    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 06:56:07.015988 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.015975    2574 policy_none.go:44] "Start"
Oct 30 06:56:07.018596 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.018569    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 06:56:07.018681 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.018666    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qqc9x"
Oct 30 06:56:07.024382 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.024360    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qqc9x"
Oct 30 06:56:07.026858 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.026834    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 06:56:07.026978 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.026877    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-124.ec2.internal.18e33c4a6f072323  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-124.ec2.internal,},FirstTimestamp:2026-10-30 06:56:07.011574563 +0000 UTC m=+0.397217235,LastTimestamp:2026-10-30 06:56:07.011574563 +0000 UTC m=+0.397217235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-124.ec2.internal,}"
Oct 30 06:56:07.047575 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.047534    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 06:56:07.047575 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.047490    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-124.ec2.internal.18e33c4a6f076f63  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-124.ec2.internal,},FirstTimestamp:2026-10-30 06:56:07.011594083 +0000 UTC m=+0.397236754,LastTimestamp:2026-10-30 06:56:07.011594083 +0000 UTC m=+0.397236754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-124.ec2.internal,}"
Oct 30 06:56:07.047784 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.047585    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 06:56:07.056047 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.055993    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 06:56:07.060450 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.060437    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 06:56:07.060531 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.060474    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 06:56:07.060531 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.060483    2574 server.go:87] "Starting device plugin registration server"
Oct 30 06:56:07.060795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.060784    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 06:56:07.060830 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.060799    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 06:56:07.060952 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.060928    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 06:56:07.061051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.061029    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 06:56:07.061051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.061049    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 06:56:07.061601 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.061580    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 06:56:07.061675 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.061636    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.137923 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.137885    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 06:56:07.139720 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.139130    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 06:56:07.139720 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.139153    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 06:56:07.139720 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.139175    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 06:56:07.139720 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.139208    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 06:56:07.152389 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.152363    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:07.161480 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.161405    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.162424 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.162406    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.162424 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.162437    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.162627 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.162449    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.162627 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.162477    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.179132 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.179107    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.179249 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.179136    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-124.ec2.internal\": node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.220418 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.220393    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.239846 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.239815    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"]
Oct 30 06:56:07.239996 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.239877    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.241871 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.241843    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.242019 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.241879    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.242019 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.241893    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.243462 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.243447    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.243627 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.243611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.243667 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.243648    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.244258 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244238    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.244381 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244268    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.244381 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244272    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.244381 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244283    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.244381 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244308    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.244381 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.244323    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.247177 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.245937    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.247177 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.245976    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:07.247649 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.247627    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:07.247742 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.247657    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:07.247742 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.247667    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:07.280198 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.280096    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-124.ec2.internal\" not found" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.283407 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.283389    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-124.ec2.internal\" not found" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.320636 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.320601    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.390584 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.390385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.390727 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.390595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.390727 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.390617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.421180 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.421116    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.491502 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.491575 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.491575 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491554    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.491647 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491570    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.491647 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.491647 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.491637    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.521659 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.521629    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.584825 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.584798    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.588083 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.587967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:07.622651 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.622613    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.724627 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.724537    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.826939 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.826900    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:07.926708 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.926678    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 06:56:07.927196 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:07.926894    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 06:56:07.927851 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:07.927832    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.028216 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.028140    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.029304 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.029271    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:07 +0000 UTC" deadline="2028-08-06 07:53:38.268532806 +0000 UTC"
Oct 30 06:56:08.029359 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.029313    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15504h57m30.239223136s"
Oct 30 06:56:08.128937 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.128893    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.162819 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.162787    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:08.164015 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.163991    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:08.164148 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.164031    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:08.164148 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.164045    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:08.186559 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.186526    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:08.204172 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:08.204133    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8de38f9da3fc95cb8d0657cd8e6c898.slice/crio-25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5 WatchSource:0}: Error finding container 25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5: Status 404 returned error can't find the container with id 25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5
Oct 30 06:56:08.206337 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:08.206305    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1bcd324f5af093ecce999f61f1d18f.slice/crio-21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363 WatchSource:0}: Error finding container 21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363: Status 404 returned error can't find the container with id 21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363
Oct 30 06:56:08.209806 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.209790    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 06:56:08.229832 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.229794    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.332791 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.332706    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.433468 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.433437    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.534379 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.534339    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.575320 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.575276    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:08.634628 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.634594    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.735386 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.735344    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.836099 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.836067    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.937236 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:08.937142    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:56:08.950048 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.950013    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:08.965149 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.965113    2574 apiserver.go:52] "Watching apiserver"
Oct 30 06:56:08.989457 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.989409    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 06:56:08.989457 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.989436    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:08.998568 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.998527    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 06:56:08.998885 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:08.998863    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gc6hx","openshift-image-registry/node-ca-nxvzr","openshift-multus/multus-9mh4r","openshift-multus/multus-additional-cni-plugins-wp2gn","openshift-network-diagnostics/network-check-target-qdhdb","openshift-network-operator/iptables-alerter-hr547","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv","openshift-dns/node-resolver-qgqkv","openshift-multus/network-metrics-daemon-vtjnb","openshift-ovn-kubernetes/ovnkube-node-6jgfh","kube-system/global-pull-secret-syncer-f2pl2","kube-system/konnectivity-agent-djbct"]
Oct 30 06:56:09.001680 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.001641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.004179 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.004150    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.006717 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.006688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.010281 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.010131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.012895 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.012597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:09.012895 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.012668    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:09.014844 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.014825    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.015955 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.015934    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:56:09.016064 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.016041    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:56:09.018192 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.018168    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.019383 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.019349    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.019626 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.019608    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2tccl\""
Oct 30 06:56:09.019699 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.019628    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.028655 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.028628    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.028796 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.028691    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.028858 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.028808    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.029110 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029081    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44sjc\""
Oct 30 06:56:09.029206 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029192    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.029306 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029269    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 06:56:09.029631 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029612    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s8tns\""
Oct 30 06:56:09.029768 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029745    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 06:56:09.029841 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029786    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 06:56:09.029923 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.029621    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 06:56:09.030017 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.030000    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.030118 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.030093    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 06:56:09.030266 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.030160    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6h5pm\""
Oct 30 06:56:09.033801 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.033749    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.033912 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.033863    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:09.036818 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.036789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.037978 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.037956    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:56:09.040802 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.040779    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.040912 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.040852    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:09.044092 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.044070    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"]
Oct 30 06:56:09.044193 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.044109    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"]
Oct 30 06:56:09.044250 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.044185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.061387 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.061344    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 06:56:09.061561 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.061431    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvf4z"
Oct 30 06:56:09.074448 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.074416    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvf4z"
Oct 30 06:56:09.090181 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.090156    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 06:56:09.101669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.101840 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.101840 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.101840 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.101840 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.102041 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.102041 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.102041 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4xp\" (UniqueName: \"kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.102041 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.101999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102041 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.102247 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.102552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.102878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.102878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.102878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.102807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.103119 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9nd\" (UniqueName: \"kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.103174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.103174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cxl\" (UniqueName: \"kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.103284 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.103284 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nj72\" (UniqueName: \"kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.103407 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.103407 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2nn\" (UniqueName: \"kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.103407 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ctg\" (UniqueName: \"kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.103407 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.103595 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.103595 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.103595 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.103762 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.103762 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.103762 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wv7\" (UniqueName: \"kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.103762 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.103909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.103909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.103909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.103909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.104078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.104078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.103978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104287 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.104614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhtn\" (UniqueName: \"kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.104962 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.104955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.105321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.105030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87gk\" (UniqueName: \"kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.105321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.105050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.105321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.105065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.105321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.105079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.127996    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128006    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128076    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-th56t\""
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128092    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128081    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.128208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128149    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.128694 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128670    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.128857 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128766    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 06:56:09.128857 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128826    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vngf\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128869    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128910    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128928    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128946    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128961    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.128992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128974    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 06:56:09.129259 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.128996    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 06:56:09.129433 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.129408    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 06:56:09.129551 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.129491    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2h44\""
Oct 30 06:56:09.129616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.129600    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 06:56:09.129658 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.129650    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7szgh\""
Oct 30 06:56:09.129729 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.129706    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q4jsx\""
Oct 30 06:56:09.145451 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.145348    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal" event={"ID":"bc1bcd324f5af093ecce999f61f1d18f","Type":"ContainerStarted","Data":"21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363"}
Oct 30 06:56:09.146889 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.146857    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerStarted","Data":"25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5"}
Oct 30 06:56:09.206321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.206321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.206321 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9nd\" (UniqueName: \"kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cxl\" (UniqueName: \"kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nj72\" (UniqueName: \"kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206549    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2nn\" (UniqueName: \"kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.206616 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.206577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ctg\" (UniqueName: \"kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207388    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207547    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207761    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wv7\" (UniqueName: \"kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.207919    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.213431 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208303    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208403    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208640    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208688    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.208959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209016    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209115    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209165    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209275    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.214439 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209341    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209386    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209438    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209483    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209669    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209721    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209743    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209774    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.215248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209818    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209827    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209968    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.209977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210002    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210273    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhtn\" (UniqueName: \"kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.215987 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210393    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87gk\" (UniqueName: \"kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210455    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210496    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210513    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210558    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210570    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.216769 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210846    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4xp\" (UniqueName: \"kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.210980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211006    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211074    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.211182    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.211264    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:09.711241713 +0000 UTC m=+3.096884399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.217523 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211325    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211449    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211585    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211660    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211870    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211913    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211947    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211978    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212008    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218230 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212152    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.211294    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212189    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212268    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212278    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212435    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212488    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212483    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.212784    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.213387    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.213490    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.213578    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:09.71355901 +0000 UTC m=+3.099201688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:09.218874 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.214116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.219558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.216436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.219558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.216805    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.233449 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.233413    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:09.233449 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.233449    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:09.233690 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.233465    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:09.233690 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.233555    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:09.733533209 +0000 UTC m=+3.119175893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:09.234539 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.234506    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2nn\" (UniqueName: \"kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.235395 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.234982    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9nd\" (UniqueName: \"kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.235499 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.235417    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nj72\" (UniqueName: \"kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.236223 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.236198    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wv7\" (UniqueName: \"kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.236308 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.236206    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cxl\" (UniqueName: \"kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.237892 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.237842    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhtn\" (UniqueName: \"kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.238742 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.238717    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4xp\" (UniqueName: \"kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.239297 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.239248    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ctg\" (UniqueName: \"kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.241856 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.241827    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87gk\" (UniqueName: \"kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.317768 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.317722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:56:09.330121 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.330064    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f96032_aac6_45a7_b77b_e7f43f33afbc.slice/crio-1aa93f1943dc3e8d75e669e3fb662a6b662213be034d94a4584788b6650e24b3 WatchSource:0}: Error finding container 1aa93f1943dc3e8d75e669e3fb662a6b662213be034d94a4584788b6650e24b3: Status 404 returned error can't find the container with id 1aa93f1943dc3e8d75e669e3fb662a6b662213be034d94a4584788b6650e24b3
Oct 30 06:56:09.335400 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.335370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:56:09.346047 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.345840    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9mh4r"
Oct 30 06:56:09.346285 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.346256    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2098dac_6268_493b_b74d_d93639ac7e8d.slice/crio-19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13 WatchSource:0}: Error finding container 19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13: Status 404 returned error can't find the container with id 19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13
Oct 30 06:56:09.356837 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.356797    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992e92c9_6236_4e46_90bb_08f2a0b89a17.slice/crio-8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e WatchSource:0}: Error finding container 8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e: Status 404 returned error can't find the container with id 8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e
Oct 30 06:56:09.367024 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.366963    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:56:09.369236 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.369205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:56:09.378924 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.378889    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:56:09.382886 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.382771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:56:09.387293 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.387268    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f128ed7_b717_4188_84f8_0ea3f320dee4.slice/crio-8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7 WatchSource:0}: Error finding container 8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7: Status 404 returned error can't find the container with id 8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7
Oct 30 06:56:09.395607 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.395576    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb0f2a4_b0bb_4259_9f93_0e2331129305.slice/crio-de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba WatchSource:0}: Error finding container de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba: Status 404 returned error can't find the container with id de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba
Oct 30 06:56:09.396177 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.396052    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6528c489_7d36_4881_a608_11ecdd2bf3f3.slice/crio-eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5 WatchSource:0}: Error finding container eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5: Status 404 returned error can't find the container with id eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5
Oct 30 06:56:09.403095 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.403068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:09.409766 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.409742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:09.415872 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.415846    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c36d37c_2e1a_41aa_9f1a_bf67e50331c9.slice/crio-da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0 WatchSource:0}: Error finding container da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0: Status 404 returned error can't find the container with id da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0
Oct 30 06:56:09.420883 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:09.420856    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb43b53_7a8a_44d5_a802_2eb2691e2758.slice/crio-a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b WatchSource:0}: Error finding container a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b: Status 404 returned error can't find the container with id a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b
Oct 30 06:56:09.475348 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.475234    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:09.715998 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.715956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:09.716175 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.716016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:09.716175 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.716133    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:09.716298 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.716228    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.716207723 +0000 UTC m=+4.101850390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:09.716298 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.716133    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:09.716419 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.716341    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.716321166 +0000 UTC m=+4.101963846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:09.816855 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:09.816796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:09.817025 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.816941    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:09.817025 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.816956    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:09.817025 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.816966    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:09.817025 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:09.817014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.817001134 +0000 UTC m=+4.202643810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:10.077540 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.077457    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:09 +0000 UTC" deadline="2028-04-05 13:48:02.510132517 +0000 UTC"
Oct 30 06:56:10.077540 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.077493    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12558h51m52.432644013s"
Oct 30 06:56:10.168895 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.168615    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gc6hx" event={"ID":"42f96032-aac6-45a7-b77b-e7f43f33afbc","Type":"ContainerStarted","Data":"1aa93f1943dc3e8d75e669e3fb662a6b662213be034d94a4584788b6650e24b3"}
Oct 30 06:56:10.179500 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.179452    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal" event={"ID":"bc1bcd324f5af093ecce999f61f1d18f","Type":"ContainerStarted","Data":"2a0780f2535339d82d1fc96ccec41a4b007d88ea1a9ea195fe18b86ef6e06dbb"}
Oct 30 06:56:10.195051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.194962    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b"}
Oct 30 06:56:10.205604 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.205525    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djbct" event={"ID":"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9","Type":"ContainerStarted","Data":"da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0"}
Oct 30 06:56:10.212919 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.212876    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qgqkv" event={"ID":"6528c489-7d36-4881-a608-11ecdd2bf3f3","Type":"ContainerStarted","Data":"eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5"}
Oct 30 06:56:10.230065 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.229995    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal" podStartSLOduration=1.229974507 podStartE2EDuration="1.229974507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:10.229236276 +0000 UTC m=+3.614878964" watchObservedRunningTime="2026-10-30 06:56:10.229974507 +0000 UTC m=+3.615617193"
Oct 30 06:56:10.230916 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.230745    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hr547" event={"ID":"2f128ed7-b717-4188-84f8-0ea3f320dee4","Type":"ContainerStarted","Data":"8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7"}
Oct 30 06:56:10.245391 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.245008    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerStarted","Data":"25211ed215be44f2f09fa6aa18d9c10188ab5f4748f3f2d69837f6f1a450885a"}
Oct 30 06:56:10.253746 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.253480    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9mh4r" event={"ID":"992e92c9-6236-4e46-90bb-08f2a0b89a17","Type":"ContainerStarted","Data":"8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e"}
Oct 30 06:56:10.267914 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.267818    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nxvzr" event={"ID":"c2098dac-6268-493b-b74d-d93639ac7e8d","Type":"ContainerStarted","Data":"19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13"}
Oct 30 06:56:10.275589 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.275557    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv" event={"ID":"5eb0f2a4-b0bb-4259-9f93-0e2331129305","Type":"ContainerStarted","Data":"de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba"}
Oct 30 06:56:10.724093 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.724042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:10.724299 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.724105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:10.724299 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.724236    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:10.724439 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.724306    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.724285395 +0000 UTC m=+6.109928065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:10.724439 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.724236    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:10.724439 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.724366    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.724352067 +0000 UTC m=+6.109994744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:10.825369 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:10.825243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:10.825559 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.825437    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:10.825559 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.825459    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:10.825559 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.825473    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:10.825559 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:10.825547    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.82552615 +0000 UTC m=+6.211168816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:11.078590 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.078466    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:09 +0000 UTC" deadline="2028-08-10 21:13:18.189001972 +0000 UTC"
Oct 30 06:56:11.078590 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.078537    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15614h17m7.110469843s"
Oct 30 06:56:11.143498 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.142652    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:11.143498 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:11.142793    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:11.143498 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.143215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:11.143498 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:11.143320    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:11.144120 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.143940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:11.144120 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:11.144044    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:11.301005 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.300949    2574 generic.go:356] "Generic (PLEG): container finished" podID="c8de38f9da3fc95cb8d0657cd8e6c898" containerID="4abded307aed34758d8f9e59db4c51f41ebc9b312a6b18d21cb560b85457d22d" exitCode=0
Oct 30 06:56:11.302261 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:11.302232    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerDied","Data":"4abded307aed34758d8f9e59db4c51f41ebc9b312a6b18d21cb560b85457d22d"}
Oct 30 06:56:12.316225 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:12.316158    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerStarted","Data":"9d3ca637e417d3d0e0b1f4d45c754a82c21e6fbadd01afa34057526912b43e65"}
Oct 30 06:56:12.748393 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:12.748346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:12.748393 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:12.748394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:12.748644 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.748511    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:12.748644 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.748563    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.748548138 +0000 UTC m=+10.134190814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:12.748945 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.748924    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.749005 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.748988    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.748972212 +0000 UTC m=+10.134614901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.850398 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:12.849640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:12.850398 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.849840    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:12.850398 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.849863    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:12.850398 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.849879    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:12.850398 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:12.849941    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.849923315 +0000 UTC m=+10.235565984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:13.139950 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.139912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:13.140169 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:13.140101    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:13.140627 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.140601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:13.140759 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:13.140731    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:13.140936 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.140917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:13.141055 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:13.141028    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:13.329191 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.327617    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="276f0323c4c47c93d90b15f7ad4372225428c8d2d8f1ba60cad2a6be865253bf" exitCode=0
Oct 30 06:56:13.329191 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.328908    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"276f0323c4c47c93d90b15f7ad4372225428c8d2d8f1ba60cad2a6be865253bf"}
Oct 30 06:56:13.412225 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:13.412121    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" podStartSLOduration=4.412105621 podStartE2EDuration="4.412105621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:12.367332381 +0000 UTC m=+5.752975068" watchObservedRunningTime="2026-10-30 06:56:13.412105621 +0000 UTC m=+6.797748307"
Oct 30 06:56:14.336453 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:14.335649    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djbct" event={"ID":"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9","Type":"ContainerStarted","Data":"4e829f0dca6b4f82e261770ea3befa4a731e28e2e42f0e3f073be340a62787cd"}
Oct 30 06:56:15.147102 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:15.147073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:15.147307 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:15.147173    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:15.147614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:15.147593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:15.147752 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:15.147726    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:15.147823 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:15.147806    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:15.147922 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:15.147901    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:15.341078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:15.341035    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nxvzr" event={"ID":"c2098dac-6268-493b-b74d-d93639ac7e8d","Type":"ContainerStarted","Data":"d8a852fe8c38a324a7d11121b089ea9d2275fb865b2579ff0b92a7f1434aa8d3"}
Oct 30 06:56:15.382105 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:15.382031    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-djbct" podStartSLOduration=4.139461408 podStartE2EDuration="8.382008046s" totalImagesPullingTime="4.242546638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:14.3827757 +0000 UTC m=+7.768418389" watchObservedRunningTime="2026-10-30 06:56:15.382008046 +0000 UTC m=+8.767650729"
Oct 30 06:56:16.345506 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:16.345467    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv" event={"ID":"5eb0f2a4-b0bb-4259-9f93-0e2331129305","Type":"ContainerStarted","Data":"45293fbd118118b1790b9e5ba5bf9345e9e722fd40188cd49fd318cdc5a935c2"}
Oct 30 06:56:16.784138 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:16.784082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:16.784894 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.784268    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:16.784894 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.784467    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.784436297 +0000 UTC m=+18.170078974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:16.785248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:16.785226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:16.785561 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.785484    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:16.785800 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.785787    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.785766603 +0000 UTC m=+18.171409279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:16.886692 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:16.886109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:16.886692 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.886305    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:16.886692 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.886328    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:16.886692 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.886339    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:16.886692 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:16.886388    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.886375319 +0000 UTC m=+18.272017995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:17.079194 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.078502    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:17.079620 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.079596    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:17.121435 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.121400    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 06:56:17.121782 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.121731    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nxvzr" podStartSLOduration=4.83914249 podStartE2EDuration="10.121717643s" totalImagesPullingTime="5.282575153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:15.383315717 +0000 UTC m=+8.768958403" watchObservedRunningTime="2026-10-30 06:56:17.121717643 +0000 UTC m=+10.507360328"
Oct 30 06:56:17.140685 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.140655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:17.140815 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:17.140788    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:17.140871 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.140858    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:17.140943 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:17.140928    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:17.141114 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.141093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:17.141229 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:17.141200    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:17.351332 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:17.351158    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv" event={"ID":"5eb0f2a4-b0bb-4259-9f93-0e2331129305","Type":"ContainerStarted","Data":"39d3866c754fdd2c2e530119dac01ab406f00d1372dfe4352985a6388e45a69b"}
Oct 30 06:56:18.067444 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.067316    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:17.121426484Z","UUID":"88386076-73d9-45cd-a0ff-1b3a92a080d9","Handler":null,"Name":"","Endpoint":""}
Oct 30 06:56:18.069594 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.069556    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 06:56:18.069594 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.069592    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 06:56:18.355990 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.355685    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv" event={"ID":"5eb0f2a4-b0bb-4259-9f93-0e2331129305","Type":"ContainerStarted","Data":"3e08bd532bef48cd76eb4cea8f0372fef9678b251e3d22a844ca1ccf558b4c8f"}
Oct 30 06:56:18.358131 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.358097    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qgqkv" event={"ID":"6528c489-7d36-4881-a608-11ecdd2bf3f3","Type":"ContainerStarted","Data":"60762e5d9f2fb8222a3f2030d815c53cfc806456d4d1c5650947a485d59a16bf"}
Oct 30 06:56:18.359756 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.359726    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="16c67563823f58d47ee989688735dbdb0219f6ce0356ecabf25db0e6bf86961f" exitCode=0
Oct 30 06:56:18.359878 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.359764    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"16c67563823f58d47ee989688735dbdb0219f6ce0356ecabf25db0e6bf86961f"}
Oct 30 06:56:18.362533 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.362482    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9mh4r" event={"ID":"992e92c9-6236-4e46-90bb-08f2a0b89a17","Type":"ContainerStarted","Data":"4964257f99a78fdba28cf2bd1ca5f9cd1ae7bcfcaea080fd3ca9ae63c9b605e5"}
Oct 30 06:56:18.402036 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.401971    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv" podStartSLOduration=3.020383213 podStartE2EDuration="11.401951649s" totalImagesPullingTime="8.381568436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:18.401059393 +0000 UTC m=+11.786702076" watchObservedRunningTime="2026-10-30 06:56:18.401951649 +0000 UTC m=+11.787594334"
Oct 30 06:56:18.446151 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.446044    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9mh4r" podStartSLOduration=3.25266091 podStartE2EDuration="11.446024845s" totalImagesPullingTime="8.193363935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:18.445938959 +0000 UTC m=+11.831581641" watchObservedRunningTime="2026-10-30 06:56:18.446024845 +0000 UTC m=+11.831667519"
Oct 30 06:56:18.536211 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:18.535722    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qgqkv" podStartSLOduration=3.109608202 podStartE2EDuration="11.535702074s" totalImagesPullingTime="8.426093872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:18.535036816 +0000 UTC m=+11.920679512" watchObservedRunningTime="2026-10-30 06:56:18.535702074 +0000 UTC m=+11.921344758"
Oct 30 06:56:19.139858 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.139678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:19.139858 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.139695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:19.139858 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:19.139810    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:19.140111 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:19.139909    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:19.140111 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.139975    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:19.140111 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:19.140052    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:19.366571 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.366477    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hr547" event={"ID":"2f128ed7-b717-4188-84f8-0ea3f320dee4","Type":"ContainerStarted","Data":"9d941e3cc73849d649b8c23ca3a22d0d2c5ef4974656ac3a2c0c90cda5eaa7f7"}
Oct 30 06:56:19.369057 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.369027    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="0af955e3058d7d5eb86df2c0466ad1c5d5a3a205c8082a48f7441c85971442c1" exitCode=0
Oct 30 06:56:19.369202 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.369151    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"0af955e3058d7d5eb86df2c0466ad1c5d5a3a205c8082a48f7441c85971442c1"}
Oct 30 06:56:19.407843 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:19.407776    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hr547" podStartSLOduration=4.593021519 podStartE2EDuration="12.407759155s" totalImagesPullingTime="7.814737636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:19.407662963 +0000 UTC m=+12.793305644" watchObservedRunningTime="2026-10-30 06:56:19.407759155 +0000 UTC m=+12.793401838"
Oct 30 06:56:20.374127 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:20.374089    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="cc7f6e1a5ca3c2ddadb4b4d335603951320427143f9f37579aa25de3fd3210f9" exitCode=0
Oct 30 06:56:20.374911 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:20.374163    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"cc7f6e1a5ca3c2ddadb4b4d335603951320427143f9f37579aa25de3fd3210f9"}
Oct 30 06:56:20.375843 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:20.375816    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gc6hx" event={"ID":"42f96032-aac6-45a7-b77b-e7f43f33afbc","Type":"ContainerStarted","Data":"916c1a8f29df55aad0f8ced821596f8a08cd2cfcb9620dca6b60c8394d2ab120"}
Oct 30 06:56:20.467422 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:20.467347    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gc6hx" podStartSLOduration=3.074710527 podStartE2EDuration="13.467327346s" totalImagesPullingTime="10.392616819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:20.466838464 +0000 UTC m=+13.852481151" watchObservedRunningTime="2026-10-30 06:56:20.467327346 +0000 UTC m=+13.852970029"
Oct 30 06:56:21.140442 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:21.140410    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:21.140642 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:21.140473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:21.140642 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:21.140472    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:21.140642 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:21.140564    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:21.140975 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:21.140937    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:21.141103 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:21.141033    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:22.387270 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:22.387070    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerStarted","Data":"dc75dc8d6fefd5446ca1f1bfa2bf3d4cec62c44b908fc0fedf3cdb99b8790591"}
Oct 30 06:56:23.140149 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.140112    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:23.140149 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.140140    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:23.140399 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.140218    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:23.140399 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:23.140353    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:23.140519 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:23.140492    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:23.140608 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:23.140592    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:23.393539 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.393453    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="dc75dc8d6fefd5446ca1f1bfa2bf3d4cec62c44b908fc0fedf3cdb99b8790591" exitCode=0
Oct 30 06:56:23.393539 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.393486    2574 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="9ae7e7babe1735fc60d0dfb10af6a241da92813289de924adac26b1286fe9c3d" exitCode=0
Oct 30 06:56:23.393951 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.393536    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"dc75dc8d6fefd5446ca1f1bfa2bf3d4cec62c44b908fc0fedf3cdb99b8790591"}
Oct 30 06:56:23.393951 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:23.393596    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerDied","Data":"9ae7e7babe1735fc60d0dfb10af6a241da92813289de924adac26b1286fe9c3d"}
Oct 30 06:56:24.399397 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.399179    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" event={"ID":"01063b0f-0dd4-40ac-abe3-edda6dcda17c","Type":"ContainerStarted","Data":"e6a09a143a93072a2b8ac9cebca7d826fcf2682b1bb1f15ff2ecdae16e2f3ba8"}
Oct 30 06:56:24.402429 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402405    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"0081968e809294316568951ce2d140e65c6aa2f425d73f7680f2694b411a04fa"}
Oct 30 06:56:24.402559 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402470    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"c2d1da3c0fec741bb9d62ad86cfa1d46cbe04fb20268699d4aed40661072c441"}
Oct 30 06:56:24.402559 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402500    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"5e8762cb7dce330980ee9927954f3cc87f9265e098af93b23d55179b65651871"}
Oct 30 06:56:24.402559 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402528    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"573ffe7d7ab95c97d6c26cf3db2c4c68329c94449ce20d1cf6e938ed5f679b64"}
Oct 30 06:56:24.402559 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402556    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"0c88c4b57addf41c2d2be9e661feba14a16613d1df1549892014c91256c6d1e7"}
Oct 30 06:56:24.402669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.402583    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"fa8c8fd6542587fdbc5c29f0167b2da77a8d6b1380da60b15fa48fa2cc0e4e11"}
Oct 30 06:56:24.475699 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.475642    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wp2gn" podStartSLOduration=7.144992025 podStartE2EDuration="17.475628869s" totalImagesPullingTime="10.330636844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:24.475337108 +0000 UTC m=+17.860979787" watchObservedRunningTime="2026-10-30 06:56:24.475628869 +0000 UTC m=+17.861271550"
Oct 30 06:56:24.848518 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.848483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:24.848518 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.848519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:24.848754 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.848647    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:24.848754 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.848691    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:24.848754 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.848733    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:40.848713171 +0000 UTC m=+34.234355832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:24.848927 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.848763    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:40.848753164 +0000 UTC m=+34.234395826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:24.949694 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:24.949654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:24.949836 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.949820    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:24.949869 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.949838    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:24.949869 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.949848    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:24.949949 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:24.949909    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:40.949894233 +0000 UTC m=+34.335536893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:25.142520 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:25.142490    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:25.142662 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:25.142566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:25.142662 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:25.142648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:25.142754 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:25.142660    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:25.142754 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:25.142733    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:25.142832 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:25.142770    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:27.142649 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:27.142620    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:27.143161 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:27.142712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:27.143161 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:27.142720    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:27.143161 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:27.142738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:27.143161 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:27.142815    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:27.143161 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:27.142880    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:27.415114 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:27.415029    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"42bcad870868caac6724c93dbbce1c99e8fe93d1b118393c96e7b113d1d2b420"}
Oct 30 06:56:28.884575 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:28.884545    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:29.028740 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.028697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:29.139986 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.139906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:29.139986 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.139934    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:29.140153 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:29.140016    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:29.140153 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.140073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:29.140153 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:29.140142    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:29.140245 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:29.140194    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:29.185730 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.185697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:29.423220 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.423145    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" event={"ID":"ffb43b53-7a8a-44d5-a802-2eb2691e2758","Type":"ContainerStarted","Data":"3cf2eea057773e1a9068e3a7df9e9cebcc00ae5878296082d7ca955ed6d33d15"}
Oct 30 06:56:29.423408 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.423388    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:29.439366 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.439192    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:29.523621 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.523570    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh" podStartSLOduration=7.917086604 podStartE2EDuration="22.523557767s" totalImagesPullingTime="14.606471163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:29.523075806 +0000 UTC m=+22.908718499" watchObservedRunningTime="2026-10-30 06:56:29.523557767 +0000 UTC m=+22.909200448"
Oct 30 06:56:29.777056 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.776985    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:29.893508 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:29.893476    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:30.425887 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.425857    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:30.426087 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.425939    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:30.441727 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.441703    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:56:30.639327 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.639274    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtjnb"]
Oct 30 06:56:30.639484 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.639474    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:30.639614 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:30.639594    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:30.641069 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.641034    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2pl2"]
Oct 30 06:56:30.641204 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.641147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:30.641273 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:30.641250    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:30.641589 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.641568    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qdhdb"]
Oct 30 06:56:30.641661 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:30.641651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:30.641728 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:30.641713    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:32.128330 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.128284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.140223 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.140197    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:32.140336 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.140304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:32.140336 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:32.140315    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:32.140421 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:32.140391    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:32.140456 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.140428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:32.140497 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:32.140484    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:32.243727 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.243694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.349815 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.349780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.364109 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.364082    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:32.364213 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.364207    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:32.364593 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.364569    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:56:32.502506 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.502433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.610808 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.610779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.720035 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.719993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.842148 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.842066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:32.954613 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:32.954566    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:33.640196 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:33.640152    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:33.816789 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:33.816728    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:34.140311 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:34.140262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:34.140311 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:34.140287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:34.140524 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:34.140386    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:34.140524 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:34.140443    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:34.140524 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:34.140463    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:34.140620 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:34.140534    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:34.490441 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:34.490408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:35.069752 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:35.069720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:35.210672 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:35.210639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:35.353556 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:35.353447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:35.532817 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:35.532787    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qgqkv_6528c489-7d36-4881-a608-11ecdd2bf3f3/dns-node-resolver/0.log"
Oct 30 06:56:35.952783 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:35.952689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:36.139457 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.139421    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:36.139885 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.139523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:36.139885 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:36.139532    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:36.139885 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:36.139622    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:36.139885 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.139534    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:36.139885 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:36.139696    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:36.531209 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.531182    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nxvzr_c2098dac-6268-493b-b74d-d93639ac7e8d/node-ca/0.log"
Oct 30 06:56:36.546809 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.546778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:36.656634 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.656599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:36.764497 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:36.764465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.161847 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.161812    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.295400 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.295367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.413596 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.413514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.555773 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.555744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.676931 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.676859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:37.976312 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:37.976208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:38.139371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:38.139340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:38.139546 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:38.139405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:38.139546 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:38.139432    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:38.139546 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:38.139518    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:38.139672 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:38.139555    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:38.139672 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:38.139621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:38.617752 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:38.617715    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:38.749078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:38.749049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:39.183009 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:39.182979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:39.350923 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:39.350889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:39.943690 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:39.943658    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:40.140141 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.140105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:40.140141 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.140126    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:40.140504 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.140220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:40.140504 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.140230    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:40.140504 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.140354    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:40.140504 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.140448    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:40.548048 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.548015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:40.748449 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.748418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:40.856058 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.856016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:40.856224 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.856065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:40.856224 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.856159    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:40.856224 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.856160    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:40.856224 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.856209    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret podName:e2ab3bbf-b449-4b3c-b048-5180eb3b2e37 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:12.856196492 +0000 UTC m=+66.241839164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret") pod "global-pull-secret-syncer-f2pl2" (UID: "e2ab3bbf-b449-4b3c-b048-5180eb3b2e37") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:40.856224 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.856227    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs podName:1d3eea62-15e2-488a-a427-abb5e81d034b nodeName:}" failed. No retries permitted until 2026-10-30 06:57:12.856217308 +0000 UTC m=+66.241859968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs") pod "network-metrics-daemon-vtjnb" (UID: "1d3eea62-15e2-488a-a427-abb5e81d034b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:40.956946 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:40.956904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:40.957354 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.957073    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:40.957354 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.957097    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:40.957354 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.957107    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9kd62 for pod openshift-network-diagnostics/network-check-target-qdhdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:40.957354 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:40.957160    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62 podName:19df0167-7313-4ba8-a63f-99ce3cd97697 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:12.957146704 +0000 UTC m=+66.342789382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9kd62" (UniqueName: "kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62") pod "network-check-target-qdhdb" (UID: "19df0167-7313-4ba8-a63f-99ce3cd97697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:41.146002 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:41.145971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:41.352654 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:41.352621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:41.628089 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:41.628055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:41.736588 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:41.736561    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:41.946343 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:41.946246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:42.057410 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.057382    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:42.140346 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.140289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:42.140533 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.140289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:42.140533 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:42.140435    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:42.140533 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.140289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:42.140533 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:42.140495    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:42.140759 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:42.140548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:42.543075 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.543045    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:42.748445 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:42.748413    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:43.145781 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:43.145751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:43.347134 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:43.347104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:43.813865 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:43.813828    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:43.951569 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:43.951537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:44.140104 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.140068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:44.140104 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.140095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:44.140325 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.140069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:44.140325 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:44.140183    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:44.140325 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:44.140261    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:44.140425 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:44.140330    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:44.552168 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.552084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:44.663614 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.663578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:44.773958 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.773926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:44.962089 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:44.962060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:45.147241 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:45.147213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:45.352579 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:45.352499    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:45.749681 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:45.749650    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:45.951050 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:45.951020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:46.140216 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:46.140183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:46.140216 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:46.140211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:46.140437 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:46.140287    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:46.140437 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:46.140346    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:46.140437 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:46.140419    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:46.140583 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:46.140559    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:46.265510 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:46.265475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:48.139593 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:48.139560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:48.139593 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:48.139588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:48.140020 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:48.139598    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:48.140020 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:48.139676    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:48.140020 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:48.139712    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:48.140020 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:48.139796    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:50.140169 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.140132    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:50.140169 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.140167    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:50.140645 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:50.140265    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:50.140645 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.140316    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:50.140645 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:50.140396    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:50.140645 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:50.140486    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:50.159660 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.159632    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:50.283132 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.283100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:50.472243 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.472171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:50.585608 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.585574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:50.697943 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:50.697908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:51.603502 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:51.602981    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:51.708600 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:51.708559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:52.140164 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.140130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:52.140389 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.140130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:52.140389 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:52.140266    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2pl2" podUID="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37"
Oct 30 06:56:52.140389 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.140130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:52.140389 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:52.140322    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdhdb" podUID="19df0167-7313-4ba8-a63f-99ce3cd97697"
Oct 30 06:56:52.140389 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:56:52.140381    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtjnb" podUID="1d3eea62-15e2-488a-a427-abb5e81d034b"
Oct 30 06:56:52.525958 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.525856    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:52.718437 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.718402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:52.848201 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.848135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:52.956784 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:52.956749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:53.366619 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.366591    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeReady"
Oct 30 06:56:53.366790 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.366718    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 06:56:53.432909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.432878    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wmj4w"]
Oct 30 06:56:53.434770 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.434750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.436302 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.436274    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rqlss"]
Oct 30 06:56:53.438236 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.438222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.459908 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.459880    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmj4w"]
Oct 30 06:56:53.467879 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.467854    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7r8w2"]
Oct 30 06:56:53.470028 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.470014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.477024 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477005    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 06:56:53.477122 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477085    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 06:56:53.477164 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477120    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 06:56:53.477164 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477149    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pptc7\""
Oct 30 06:56:53.477229 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477178    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5k2w\""
Oct 30 06:56:53.477229 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477179    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 06:56:53.477229 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477208    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 06:56:53.477229 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.477213    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 06:56:53.491510 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.491482    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rqlss"]
Oct 30 06:56:53.492004 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.491981    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7r8w2"]
Oct 30 06:56:53.542051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542019    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.542051 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.542259 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.542259 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8ls\" (UniqueName: \"kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.542259 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.542372 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.542372 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.542372 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgc4\" (UniqueName: \"kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.542459 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.542452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.575339 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.575282    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 06:56:53.575339 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.575288    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 06:56:53.575528 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.575351    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 06:56:53.575528 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.575441    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dpx9k\""
Oct 30 06:56:53.643808 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.643960 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.643960 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.643960 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8ls\" (UniqueName: \"kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.643960 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.643960 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.644178 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.643987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.644178 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgc4\" (UniqueName: \"kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.644178 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.644178 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9fj\" (UniqueName: \"kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.644178 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.644456 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644315    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.644456 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644379    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.644456 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644447    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.644611 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.644518    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.645539 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.645517    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.648570 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.648553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.648629 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.648594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.668340 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.668315    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgc4\" (UniqueName: \"kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.668470 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.668431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8ls\" (UniqueName: \"kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.745467 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.745434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.745467 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.745473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9fj\" (UniqueName: \"kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.747581 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.747560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:53.747810 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.747791    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.753517 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.753489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:56:53.770138 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.770111    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9fj\" (UniqueName: \"kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:53.779925 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:53.779882    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:56:54.060227 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.060206    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rqlss"]
Oct 30 06:56:54.071613 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.070762    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmj4w"]
Oct 30 06:56:54.073208 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:54.073180    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de252f7_3dda_4a15_abb2_a69f2b22fb7e.slice/crio-53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9 WatchSource:0}: Error finding container 53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9: Status 404 returned error can't find the container with id 53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9
Oct 30 06:56:54.094691 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.094671    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7r8w2"]
Oct 30 06:56:54.096113 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:56:54.096093    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f67d3c_20ef_48ee_b6a5_c33e841afd3d.slice/crio-7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc WatchSource:0}: Error finding container 7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc: Status 404 returned error can't find the container with id 7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc
Oct 30 06:56:54.140172 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.140151    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:56:54.140255 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.140197    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:56:54.140326 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.140312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:56:54.156638 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.156611    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:56:54.156899 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.156880    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:56:54.157009 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.156982    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:56:54.157069 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.157004    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:56:54.157069 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.157035    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:56:54.157163 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.157078    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:56:54.157249 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.157231    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:56:54.479388 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.479353    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7r8w2" event={"ID":"10f67d3c-20ef-48ee-b6a5-c33e841afd3d","Type":"ContainerStarted","Data":"7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc"}
Oct 30 06:56:54.481083 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.481058    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqlss" event={"ID":"358670d6-0bd0-4c3b-81cb-beafe2872176","Type":"ContainerStarted","Data":"894e564d1128847853a49a264df75030a481929b1c431610f37fb34c9c43d9db"}
Oct 30 06:56:54.481209 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.481110    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqlss" event={"ID":"358670d6-0bd0-4c3b-81cb-beafe2872176","Type":"ContainerStarted","Data":"bcf73f3bf23b6fac430bf4052608737530ef132e2f82fe992fbbcbbf17dc7860"}
Oct 30 06:56:54.482246 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:54.482222    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmj4w" event={"ID":"7de252f7-3dda-4a15-abb2-a69f2b22fb7e","Type":"ContainerStarted","Data":"53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9"}
Oct 30 06:56:55.489445 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:55.489407    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqlss" event={"ID":"358670d6-0bd0-4c3b-81cb-beafe2872176","Type":"ContainerStarted","Data":"88990712fb94f5bbda1b760cee7eab0e033b423939a7c8804728bc5f64f0cf03"}
Oct 30 06:56:55.782927 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:55.782782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qgqkv_6528c489-7d36-4881-a608-11ecdd2bf3f3/dns-node-resolver/0.log"
Oct 30 06:56:56.164558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.164441    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:56.276322 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.276261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:56.391266 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.391232    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:56.494430 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.494334    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqlss" event={"ID":"358670d6-0bd0-4c3b-81cb-beafe2872176","Type":"ContainerStarted","Data":"eb38517b4a19a690f226b97a159a40c2b17cc5831d5f1410fe808a599657dc37"}
Oct 30 06:56:56.496420 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.496385    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmj4w" event={"ID":"7de252f7-3dda-4a15-abb2-a69f2b22fb7e","Type":"ContainerStarted","Data":"bab338fe21bb3d360bdbdeec7e88c44632d3035e103a86653a1404e06e3be33b"}
Oct 30 06:56:56.496554 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.496439    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmj4w" event={"ID":"7de252f7-3dda-4a15-abb2-a69f2b22fb7e","Type":"ContainerStarted","Data":"d6b597a8eb14fddca34d1fdec2622973ffb98ef6ff93f25942ff7f268dcf863e"}
Oct 30 06:56:56.496554 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.496496    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:56:56.496791 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.496771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:56.540786 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.540730    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rqlss" podStartSLOduration=1.589139023 podStartE2EDuration="3.540715719s" totalImagesPullingTime="1.951576696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:56.540344618 +0000 UTC m=+49.925987301" watchObservedRunningTime="2026-10-30 06:56:56.540715719 +0000 UTC m=+49.926358401"
Oct 30 06:56:56.586543 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.586477    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wmj4w" podStartSLOduration=1.846839006 podStartE2EDuration="3.586455731s" totalImagesPullingTime="1.739616725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:56.585491058 +0000 UTC m=+49.971133741" watchObservedRunningTime="2026-10-30 06:56:56.586455731 +0000 UTC m=+49.972098414"
Oct 30 06:56:56.983208 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:56.983180    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nxvzr_c2098dac-6268-493b-b74d-d93639ac7e8d/node-ca/0.log"
Oct 30 06:56:57.330550 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.330512    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:57.461598 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.461567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:57.500558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.500522    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7r8w2" event={"ID":"10f67d3c-20ef-48ee-b6a5-c33e841afd3d","Type":"ContainerStarted","Data":"3759e89bc700dfbe793739a5fe1c7480d438af03ca6cd8caaf92f2153d9be816"}
Oct 30 06:56:57.591609 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.591527    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:57.597270 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.597223    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7r8w2" podStartSLOduration=1.970854497 podStartE2EDuration="4.597208688s" totalImagesPullingTime="2.626354191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:57.596675325 +0000 UTC m=+50.982318008" watchObservedRunningTime="2026-10-30 06:56:57.597208688 +0000 UTC m=+50.982851367"
Oct 30 06:56:57.704854 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.704816    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:57.813232 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:57.813206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:56:58.747875 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:56:58.747838    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:00.674697 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:00.674661    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:00.832018 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:00.831983    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:00.981729 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:00.981642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.101054 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.101023    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"]
Oct 30 06:57:01.103099 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.103079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.124556 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.124529    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 06:57:01.126697 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.126670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.153513 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.153475    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 06:57:01.153699 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.153652    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 06:57:01.155024 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.155005    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 06:57:01.157378 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.157352    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"]
Oct 30 06:57:01.157521 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.157417    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"]
Oct 30 06:57:01.160395 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.160371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.167472 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.167450    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 06:57:01.167584 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.167491    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-75crj\""
Oct 30 06:57:01.168407 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.168387    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vns7r"]
Oct 30 06:57:01.170476 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.170459    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"]
Oct 30 06:57:01.170565 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.170556    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203368 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.203368 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.203605 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203605 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203605 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203605 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203605 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.203795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqjh\" (UniqueName: \"kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203672    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.203976 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203976 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203976 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78hq\" (UniqueName: \"kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.203976 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctkj\" (UniqueName: \"kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.203976 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.204174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.203998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.204174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.204014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.204174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.204083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.224979 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.224945    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 06:57:01.225164 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.225149    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 06:57:01.238986 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.238899    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 06:57:01.238986 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.238951    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2n2c2\""
Oct 30 06:57:01.238986 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.238967    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 06:57:01.239174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.239008    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djnbg\""
Oct 30 06:57:01.239174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.239015    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 06:57:01.239174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.239055    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 06:57:01.242081 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.242060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.304448 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.304630 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.304630 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.304739 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.304739 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.304850 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78hq\" (UniqueName: \"kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.304850 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctkj\" (UniqueName: \"kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.304850 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305000 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305000 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304908    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305000 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304928    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.305000 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.304958    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305184 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305184 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305184 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.305184 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305207    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.305414 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqjh\" (UniqueName: \"kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305692 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305459    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.305692 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305492    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.305793 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.305911 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.305888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.306043 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.306013    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.306109 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.306046    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.306479 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.306452    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.307674 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.307649    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.308330 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.308278    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.308430 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.308347    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.308473 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.308454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.308662 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.308633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.308697 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.308635    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.332426 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.332396    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78hq\" (UniqueName: \"kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.332552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.332439    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqjh\" (UniqueName: \"kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.332552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.332463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctkj\" (UniqueName: \"kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.351836 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.351803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.414722 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.414688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:01.460042 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.459770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.473362 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.473331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:01.482776 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.482747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:01.492989 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:01.492950    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6830ae7_bf50_490b_bbf8_f5d32835edfb.slice/crio-b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca WatchSource:0}: Error finding container b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca: Status 404 returned error can't find the container with id b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca
Oct 30 06:57:01.510764 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.510724    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vns7r" event={"ID":"f6830ae7-bf50-490b-bbf8-f5d32835edfb","Type":"ContainerStarted","Data":"b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca"}
Oct 30 06:57:01.616310 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.616266    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.719727 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.719702    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"]
Oct 30 06:57:01.721807 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:01.721782    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02f2149_de58_41d0_892a_77532f24a26b.slice/crio-0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d WatchSource:0}: Error finding container 0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d: Status 404 returned error can't find the container with id 0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d
Oct 30 06:57:01.752794 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.752763    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:01.771194 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.771168    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"]
Oct 30 06:57:01.772226 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:01.772203    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2a8f9c_d148_4051_855b_79b8da1b57dc.slice/crio-334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9 WatchSource:0}: Error finding container 334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9: Status 404 returned error can't find the container with id 334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9
Oct 30 06:57:01.958127 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:01.958093    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.079746 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.079675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.190729 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.190696    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.349714 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.349683    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.442588 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.442557    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:02.461127 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.461095    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:02.465221 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.465194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.465370 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.465204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.496358 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.496265    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 06:57:02.498160 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498135    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:02.498546 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498518    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 06:57:02.498651 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498600    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 06:57:02.498705 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498662    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 06:57:02.498781 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498766    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 06:57:02.498883 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498836    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 06:57:02.498941 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498899    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 06:57:02.498991 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.498961    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 06:57:02.499037 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.499004    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 06:57:02.499112 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.499096    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b8vbj\""
Oct 30 06:57:02.517912 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.517272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518047 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.517956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518047 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.517999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518047 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518205 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518205 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518146    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518317 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518375 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518375 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518460 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518392    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518460 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518544 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.518595 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.518560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.522678 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.522648    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk" event={"ID":"0f2a8f9c-d148-4051-855b-79b8da1b57dc","Type":"ContainerStarted","Data":"334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9"}
Oct 30 06:57:02.524737 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.524716    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vns7r" event={"ID":"f6830ae7-bf50-490b-bbf8-f5d32835edfb","Type":"ContainerStarted","Data":"cfd7637b833ee560b15dd59911a7df0d6e991b74f515369231613ce5d2fe443b"}
Oct 30 06:57:02.526977 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.526945    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z" event={"ID":"f02f2149-de58-41d0-892a-77532f24a26b","Type":"ContainerStarted","Data":"00cf852ea2720e84c8f3f6864d83279e5ae05f5a372ed90de26f67df773f34f8"}
Oct 30 06:57:02.527078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.527000    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z" event={"ID":"f02f2149-de58-41d0-892a-77532f24a26b","Type":"ContainerStarted","Data":"3fd10747bda369dce80e02ec5dbdc5abd53b602d7d730c529cb59e923b1dc99c"}
Oct 30 06:57:02.527078 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.527026    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z" event={"ID":"f02f2149-de58-41d0-892a-77532f24a26b","Type":"ContainerStarted","Data":"0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d"}
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.619751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.619829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.619858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.619918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.619972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620123 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620669 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.620544    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.620964 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:57:02.620878    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle podName:fa866eab-cad8-49ed-9c13-52a4f33605a8 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:03.120858144 +0000 UTC m=+56.506500822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8") : configmap references non-existent config key: ca-bundle.crt
Oct 30 06:57:02.621609 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.621582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.623579 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.623178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.624272 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.624228    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.624489 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.624469    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.625007 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.624979    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.625167 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.625145    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.625339 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.625311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.626231 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.626201    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.626512 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.626491    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.627495 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.627459    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.650557 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.650512    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:02.815968 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.815938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:02.925174 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:02.925137    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:03.125736 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.125710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:03.125947 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:57:03.125914    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle podName:fa866eab-cad8-49ed-9c13-52a4f33605a8 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:04.125888834 +0000 UTC m=+57.511531515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8") : configmap references non-existent config key: ca-bundle.crt
Oct 30 06:57:03.127644 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.127607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:03.325213 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.325176    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86b57b86db-sjlhb"]
Oct 30 06:57:03.328172 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.328149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.348435 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348405    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348475    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4pt2f\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348490    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fqkebqo6dt9ij\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348502    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348497    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348502    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 06:57:03.348586 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.348550    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 06:57:03.355285 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.355258    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b57b86db-sjlhb"]
Oct 30 06:57:03.429410 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429410 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429410 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429392    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429612 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrnm\" (UniqueName: \"kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429612 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429676 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429676 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429755 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.429795 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.429782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530173 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530173 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrnm\" (UniqueName: \"kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.530944 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.530742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.531177 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.531155    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.533404 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.532719    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk" event={"ID":"0f2a8f9c-d148-4051-855b-79b8da1b57dc","Type":"ContainerStarted","Data":"f089036453195559fb8af4ad429985de813200cc87737146335bc28d4d106db1"}
Oct 30 06:57:03.533404 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.532782    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk" event={"ID":"0f2a8f9c-d148-4051-855b-79b8da1b57dc","Type":"ContainerStarted","Data":"581da4e9b09b23b8c4e77fdb1b184ec8cab9c0df8a618cfa1969a73f2169a475"}
Oct 30 06:57:03.533404 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.532810    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk" event={"ID":"0f2a8f9c-d148-4051-855b-79b8da1b57dc","Type":"ContainerStarted","Data":"066e38fb6a854ab61726d26633d70c4b34cbf56a52343aea82ed3ea5d1c07953"}
Oct 30 06:57:03.533631 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.533438    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.533983 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.533944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.533983 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.533977    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.534111 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.534005    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.534176 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.534152    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.534376 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.534352    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.535571 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.535544    2574 generic.go:356] "Generic (PLEG): container finished" podID="f6830ae7-bf50-490b-bbf8-f5d32835edfb" containerID="cfd7637b833ee560b15dd59911a7df0d6e991b74f515369231613ce5d2fe443b" exitCode=0
Oct 30 06:57:03.535649 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.535626    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vns7r" event={"ID":"f6830ae7-bf50-490b-bbf8-f5d32835edfb","Type":"ContainerDied","Data":"cfd7637b833ee560b15dd59911a7df0d6e991b74f515369231613ce5d2fe443b"}
Oct 30 06:57:03.535710 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.535696    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vns7r" event={"ID":"f6830ae7-bf50-490b-bbf8-f5d32835edfb","Type":"ContainerStarted","Data":"748639fbf1bebe52dacd20fb34d36d5ac65db35fba195649e186c1a4140ee4bd"}
Oct 30 06:57:03.535759 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.535724    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vns7r" event={"ID":"f6830ae7-bf50-490b-bbf8-f5d32835edfb","Type":"ContainerStarted","Data":"a0d287dbea049ba53890b851b0c0360694bde336f594732ca7d49eec74e0822d"}
Oct 30 06:57:03.537979 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.537952    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z" event={"ID":"f02f2149-de58-41d0-892a-77532f24a26b","Type":"ContainerStarted","Data":"57d8073b00eed1cd9c49f0e060a37df126e6b400033ae86798af225764df961b"}
Oct 30 06:57:03.554906 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.554872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrnm\" (UniqueName: \"kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.580170 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.580111    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk" podStartSLOduration=1.340120807 podStartE2EDuration="2.580097213s" totalImagesPullingTime="1.239976406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:03.578647014 +0000 UTC m=+56.964289697" watchObservedRunningTime="2026-10-30 06:57:03.580097213 +0000 UTC m=+56.965739913"
Oct 30 06:57:03.589602 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.589567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:03.624071 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.624014    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z" podStartSLOduration=1.699785812 podStartE2EDuration="2.624002666s" totalImagesPullingTime="924.216854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:03.622985347 +0000 UTC m=+57.008628030" watchObservedRunningTime="2026-10-30 06:57:03.624002666 +0000 UTC m=+57.009645393"
Oct 30 06:57:03.649082 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.649055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:03.671511 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.671463    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vns7r" podStartSLOduration=1.729500582 podStartE2EDuration="2.671446826s" totalImagesPullingTime="855.891908ms" totalInitContainerRuntime="86.054336ms" isStatefulPod=true podCreationTimestamp="2026-10-30 06:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:03.670861886 +0000 UTC m=+57.056504572" watchObservedRunningTime="2026-10-30 06:57:03.671446826 +0000 UTC m=+57.057089498"
Oct 30 06:57:03.705783 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.705745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:03.962417 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.962319    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:03.975073 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:03.975045    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b57b86db-sjlhb"]
Oct 30 06:57:03.987958 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:03.987933    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7eb1ed_eaa2_40cb_b4a6_92d6becc8d91.slice/crio-91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419 WatchSource:0}: Error finding container 91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419: Status 404 returned error can't find the container with id 91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419
Oct 30 06:57:04.078718 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.078688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:04.136274 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.136243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:04.136449 ip-10-0-139-124 kubenswrapper[2574]: E1030 06:57:04.136394    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle podName:fa866eab-cad8-49ed-9c13-52a4f33605a8 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:06.136377888 +0000 UTC m=+59.522020548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8") : configmap references non-existent config key: ca-bundle.crt
Oct 30 06:57:04.329418 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.329343    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:04.442232 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.442191    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:04.542774 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.542663    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419"}
Oct 30 06:57:04.649200 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:04.649167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:05.039026 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.038943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:05.171434 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.171397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:05.831185 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.831147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:05.853036 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.853008    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"]
Oct 30 06:57:05.855175 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.855159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:05.878746 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.878720    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"]
Oct 30 06:57:05.927863 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.927840    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 06:57:05.927999 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.927966    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ldd2l\""
Oct 30 06:57:05.955309 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.955257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bfr\" (UniqueName: \"kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:05.955502 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:05.955317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.032225 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.032143    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:06.056558 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.056526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bfr\" (UniqueName: \"kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.056867 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.056565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.058955 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.058932    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.084520 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.084481    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bfr\" (UniqueName: \"kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.158495 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.158441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:06.159626 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.159523    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:06.166279 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.166257    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:06.198501 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.198455    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:06.391227 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.391186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:06.469737 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.469588    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"]
Oct 30 06:57:06.470897 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:06.470865    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f2a963_5150_4193_826b_bfd5a336ce5b.slice/crio-85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68 WatchSource:0}: Error finding container 85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68: Status 404 returned error can't find the container with id 85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68
Oct 30 06:57:06.503245 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.503217    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:06.558072 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.558030    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"c045792671a0077a4df6a0400455fee39626f4ccd1bd9512751e542d647c0846"}
Oct 30 06:57:06.558251 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.558158    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"e2a431eea2e39de2a79899f45dad61e7ad104807cc456a6ba3aad75f13dc033a"}
Oct 30 06:57:06.558251 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.558204    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"91d603131a6c11f87662c282ec1c312d41bb2fab2c5f5a645f1360fb091f4b3e"}
Oct 30 06:57:06.559452 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.559410    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj" event={"ID":"17f2a963-5150-4193-826b-bfd5a336ce5b","Type":"ContainerStarted","Data":"85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68"}
Oct 30 06:57:06.717748 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:06.717704    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:06.720172 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:06.720144    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa866eab_cad8_49ed_9c13_52a4f33605a8.slice/crio-6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763 WatchSource:0}: Error finding container 6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763: Status 404 returned error can't find the container with id 6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763
Oct 30 06:57:07.562619 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.562578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:07.564576 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.564546    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763"}
Oct 30 06:57:07.567761 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.567729    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"666f42e695e5c0d8d295e7786db4afbd9cd38d3fc8f8a65c374929b2564183c9"}
Oct 30 06:57:07.567879 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.567791    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"64828082d1462bb97f2dee7bf831c77db35a3444128e347663562cf9c4d2c4e0"}
Oct 30 06:57:07.567879 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.567817    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" event={"ID":"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91","Type":"ContainerStarted","Data":"0886ae82e25654df6ce4a858c5741f553da7856170556ff7f5802ffc81d37bac"}
Oct 30 06:57:07.567992 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.567896    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:07.648909 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.648627    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb" podStartSLOduration=2.231248442 podStartE2EDuration="4.648608827s" totalImagesPullingTime="2.417360385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:07.633949035 +0000 UTC m=+61.019591719" watchObservedRunningTime="2026-10-30 06:57:07.648608827 +0000 UTC m=+61.034251509"
Oct 30 06:57:07.670954 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.670930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:07.995548 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:07.995514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:08.572067 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.572034    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a" exitCode=0
Oct 30 06:57:08.572498 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.572119    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a"}
Oct 30 06:57:08.573709 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.573688    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj" event={"ID":"17f2a963-5150-4193-826b-bfd5a336ce5b","Type":"ContainerStarted","Data":"50111cc22617e88bb4bbff6bbe5f09dff741c1605f78cf72462ca879f544565a"}
Oct 30 06:57:08.574086 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.574064    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:08.578856 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.578835    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:08.677937 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:08.677884    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj" podStartSLOduration=2.537863041 podStartE2EDuration="3.677868726s" totalImagesPullingTime="1.140005685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:08.676710183 +0000 UTC m=+62.062352867" watchObservedRunningTime="2026-10-30 06:57:08.677868726 +0000 UTC m=+62.063511408"
Oct 30 06:57:09.979971 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:09.979938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:10.584197 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584105    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"}
Oct 30 06:57:10.584371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584200    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"}
Oct 30 06:57:10.584371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584229    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"}
Oct 30 06:57:10.584371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584253    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"}
Oct 30 06:57:10.584371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584277    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"}
Oct 30 06:57:10.584371 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.584319    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerStarted","Data":"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"}
Oct 30 06:57:10.684337 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.684270    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.427126947 podStartE2EDuration="8.684255673s" totalImagesPullingTime="2.243125702s" totalInitContainerRuntime="14.003024ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:10.669372719 +0000 UTC m=+64.055015401" watchObservedRunningTime="2026-10-30 06:57:10.684255673 +0000 UTC m=+64.069898354"
Oct 30 06:57:10.978182 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.978147    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:10.980735 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:10.980716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.002864 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.002834    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 06:57:11.003077 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.003060    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 06:57:11.003077 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.003060    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 06:57:11.003168 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.003077    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 06:57:11.003168 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.003077    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 06:57:11.017664 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.017637    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 06:57:11.018770 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.018752    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:11.077526 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.077486    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 06:57:11.077703 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.077602    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vjmtx\""
Oct 30 06:57:11.077703 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.077608    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 06:57:11.103147 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103147 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103419 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103419 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103419 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103532 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103450    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103532 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.103532 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.103513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204497 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204497 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204760 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204760 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204760 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204760 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204937 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.204937 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.205029 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.204986    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.205317 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.205266    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.205429 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.205344    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.205566 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.205541    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.205894 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.205875    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.207248 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.207230    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.207488 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.207466    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.230097 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.230028    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.291496 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.291456    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:11.597588 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.597559    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:11.598475 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:11.598451    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f664d3f_d782_4676_8de4_0dbd66f2d5c6.slice/crio-1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1 WatchSource:0}: Error finding container 1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1: Status 404 returned error can't find the container with id 1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1
Oct 30 06:57:11.919552 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:11.919520    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:12.124981 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.124934    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:12.592423 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.592390    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666d7bbf55-jm86p" event={"ID":"2f664d3f-d782-4676-8de4-0dbd66f2d5c6","Type":"ContainerStarted","Data":"1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1"}
Oct 30 06:57:12.921754 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.921711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:12.921943 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.921785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:12.937114 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.937083    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:57:12.944683 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.944657    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:12.950877 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.950845    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:57:12.954329 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:12.954289    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:13.022951 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.022908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:13.038193 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.038160    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:57:13.058309 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.058258    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:57:13.066202 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.066172    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:13.068599 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.068567    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:57:13.073759 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.073738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:13.077585 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.077562    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:57:13.082757 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.082734    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:13.083495 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.083469    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:57:13.088773 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.088749    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:13.384658 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.384613    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2pl2"]
Oct 30 06:57:13.385847 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:13.385815    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ab3bbf_b449_4b3c_b048_5180eb3b2e37.slice/crio-cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34 WatchSource:0}: Error finding container cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34: Status 404 returned error can't find the container with id cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34
Oct 30 06:57:13.415094 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.414789    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtjnb"]
Oct 30 06:57:13.420621 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.420596    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qdhdb"]
Oct 30 06:57:13.420923 ip-10-0-139-124 kubenswrapper[2574]: W1030 06:57:13.420896    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19df0167_7313_4ba8_a63f_99ce3cd97697.slice/crio-012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569 WatchSource:0}: Error finding container 012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569: Status 404 returned error can't find the container with id 012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569
Oct 30 06:57:13.583351 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.583322    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:13.604045 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.604012    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qdhdb" event={"ID":"19df0167-7313-4ba8-a63f-99ce3cd97697","Type":"ContainerStarted","Data":"012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569"}
Oct 30 06:57:13.605550 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.605495    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtjnb" event={"ID":"1d3eea62-15e2-488a-a427-abb5e81d034b","Type":"ContainerStarted","Data":"188aa4ebac4aeea496b6b15daa92541f2fcee0ad44d46122bee97e3e42c717f5"}
Oct 30 06:57:13.607743 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:13.606833    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2pl2" event={"ID":"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37","Type":"ContainerStarted","Data":"cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34"}
Oct 30 06:57:14.896997 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:14.896897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:15.014862 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.014824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:15.125504 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.125334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:15.234080 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.234006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:15.631658 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.631618    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtjnb" event={"ID":"1d3eea62-15e2-488a-a427-abb5e81d034b","Type":"ContainerStarted","Data":"6114025fffefee3a52a7bc7bfe2e09fdf3ef5607e461ff6734e145d037efa827"}
Oct 30 06:57:15.631816 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.631690    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtjnb" event={"ID":"1d3eea62-15e2-488a-a427-abb5e81d034b","Type":"ContainerStarted","Data":"ea2f1c32f59675a6840388d0f67400cde80089b4ed1f6978297e0b6c8865adb3"}
Oct 30 06:57:15.636227 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.636168    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666d7bbf55-jm86p" event={"ID":"2f664d3f-d782-4676-8de4-0dbd66f2d5c6","Type":"ContainerStarted","Data":"29697f8de841c49547738e68dd8be773a9c031e3c39a2fbf74876cf8bba60f5b"}
Oct 30 06:57:15.728033 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.727497    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vtjnb" podStartSLOduration=67.208696314 podStartE2EDuration="1m8.727482908s" totalImagesPullingTime="1.518786594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:15.679558967 +0000 UTC m=+69.065201674" watchObservedRunningTime="2026-10-30 06:57:15.727482908 +0000 UTC m=+69.113125590"
Oct 30 06:57:15.728033 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.727849    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-666d7bbf55-jm86p" podStartSLOduration=1.9016792329999999 podStartE2EDuration="5.727842015s" totalImagesPullingTime="3.826162782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:15.726620267 +0000 UTC m=+69.112262950" watchObservedRunningTime="2026-10-30 06:57:15.727842015 +0000 UTC m=+69.113484697"
Oct 30 06:57:15.814236 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.814153    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:15.942854 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:15.942813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:16.065277 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:16.065206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:17.392728 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:17.392684    2574 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:17.791465 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:17.791423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:17.910125 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:17.910087    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:18.018847 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.018767    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:18.147025 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.146986    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:18.365633 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.365548    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:18.472311 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.472262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:18.650017 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.649980    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qdhdb" event={"ID":"19df0167-7313-4ba8-a63f-99ce3cd97697","Type":"ContainerStarted","Data":"60187cc63df8f57b3df3bf7d4fb809baa60e9dee38d8e5ae1903550a8c5b632a"}
Oct 30 06:57:18.650273 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.650253    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:18.693881 ip-10-0-139-124 kubenswrapper[2574]: I1030 06:57:18.693821    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qdhdb" podStartSLOduration=67.452817976 podStartE2EDuration="1m11.693802169s" totalImagesPullingTime="4.240984193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:18.693458949 +0000 UTC m=+72.079101631" watchObservedRunningTime="2026-10-30 06:57:18.693802169 +0000 UTC m=+72.079444850"
Oct 30 06:57:19.501831 ip-10-0-139-124 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 06:57:19.523752 ip-10-0-139-124 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 06:57:19.523923 ip-10-0-139-124 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 06:57:19.523973 ip-10-0-139-124 systemd[1]: kubelet.service: Consumed 4.489s CPU time.
Oct 30 06:57:19.535093 ip-10-0-139-124 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 06:57:19.598237 ip-10-0-139-124 kubenswrapper[6738]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:19.598237 ip-10-0-139-124 kubenswrapper[6738]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 06:57:19.598237 ip-10-0-139-124 kubenswrapper[6738]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:19.598237 ip-10-0-139-124 kubenswrapper[6738]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600280    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600386    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600391    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600394    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600398    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.600390 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600401    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600404    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600407    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600420    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600424    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600426    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600429    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600432    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600435    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600437    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600440    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600443    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600446    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600449    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600451    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600454    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600456    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600459    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600462    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.600618 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600466    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600470    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600474    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600477    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600480    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600482    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600485    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600488    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600490    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600494    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600496    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600499    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600501    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600505    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600508    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600510    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600513    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600516    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600519    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.601079 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600522    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600525    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600527    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600530    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600533    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600535    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600538    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600540    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600543    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600545    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600548    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600551    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600554    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600557    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600560    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600562    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600565    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600568    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600570    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600573    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.601568 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600575    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600578    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600581    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600584    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600586    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600589    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600593    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600595    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600598    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600601    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600604    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600607    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600610    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600612    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600615    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600617    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600620    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600623    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600625    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600628    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.602196 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600630    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600633    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600636    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600641    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600645    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600648    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600651    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600653    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600656    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600658    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600661    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600663    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600666    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600669    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600672    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600674    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600685    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600689    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600692    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.603008 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600695    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600697    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600700    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.600704    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601607    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601616    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601620    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601622    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601625    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601629    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601631    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601634    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601636    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601639    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601642    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601644    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601647    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601649    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601652    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601655    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.603784 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601658    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601660    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601666    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601669    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601672    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601675    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601677    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601680    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601683    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601686    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601688    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601691    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601693    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601696    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601699    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601701    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601712    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601714    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601717    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601719    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.604606 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601722    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601725    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601728    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601730    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601733    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601735    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601738    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601741    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601743    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601746    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601748    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601751    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601754    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601756    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601761    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601765    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601770    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601773    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601776    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.605388 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601779    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601782    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601786    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601790    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601792    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601795    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601797    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601800    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601803    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601806    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601814    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601817    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601819    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601822    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601825    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601827    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601830    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601832    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601835    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601837    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.605907 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601840    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601842    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601845    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601847    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601850    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601852    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601855    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601859    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601861    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601864    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601866    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601869    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601872    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601874    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601877    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601880    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601882    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601885    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601889    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601891    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.606426 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601894    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601896    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601899    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601907    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601910    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601912    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601915    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601918    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601920    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601923    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.601926    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602472    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602480    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602483    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602486    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602489    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602492    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602496    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602498    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602504    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.607011 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602506    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602509    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602512    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602515    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602517    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602520    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602522    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602525    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602528    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602530    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602533    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602536    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602538    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602541    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602544    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602552    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602555    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602557    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602560    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602562    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.607823 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602565    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602568    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602572    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602575    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602577    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602580    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602582    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602585    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602587    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602590    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602592    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602596    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602599    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602602    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602604    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602607    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602610    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602613    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602616    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.608578 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602618    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602621    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602623    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602626    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602629    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602631    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602635    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602638    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602641    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602649    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602652    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602655    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602657    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602660    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602662    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602666    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602668    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602671    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602673    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.609243 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602676    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602678    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602681    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602683    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602686    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602690    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602694    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602698    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602701    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602703    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602706    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602709    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602712    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602714    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602717    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602719    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602722    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602724    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602727    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.609721 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602730    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602732    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602735    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602737    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602747    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602750    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602752    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602755    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602757    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602761    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602764    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602766    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602769    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602771    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602774    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602776    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602779    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602782    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602786    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.602789    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.610180 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602891    6738 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602909    6738 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602919    6738 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602926    6738 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602930    6738 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602934    6738 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602939    6738 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602948    6738 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602951    6738 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602954    6738 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602958    6738 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602961    6738 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602964    6738 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602968    6738 flags.go:64] FLAG: --cgroup-root=""
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602971    6738 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602974    6738 flags.go:64] FLAG: --client-ca-file=""
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602977    6738 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602986    6738 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602993    6738 flags.go:64] FLAG: --cluster-domain=""
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602996    6738 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.602999    6738 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603004    6738 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603007    6738 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603012    6738 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 06:57:19.610710 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603015    6738 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603018    6738 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603022    6738 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603025    6738 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603028    6738 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603031    6738 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603036    6738 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603040    6738 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603044    6738 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603047    6738 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603050    6738 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603053    6738 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603057    6738 flags.go:64] FLAG: --enable-server="true"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603059    6738 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603066    6738 flags.go:64] FLAG: --event-burst="100"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603069    6738 flags.go:64] FLAG: --event-qps="50"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603073    6738 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603076    6738 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603080    6738 flags.go:64] FLAG: --eviction-hard=""
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603084    6738 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603087    6738 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603090    6738 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603094    6738 flags.go:64] FLAG: --eviction-soft=""
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603097    6738 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603100    6738 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 06:57:19.611275 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603103    6738 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603112    6738 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603115    6738 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603118    6738 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603122    6738 flags.go:64] FLAG: --feature-gates=""
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603127    6738 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603130    6738 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603133    6738 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603137    6738 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603141    6738 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603144    6738 flags.go:64] FLAG: --help="false"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603148    6738 flags.go:64] FLAG: --hostname-override="ip-10-0-139-124.ec2.internal"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603151    6738 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603156    6738 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603159    6738 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603163    6738 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603166    6738 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603169    6738 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603172    6738 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603175    6738 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603178    6738 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603182    6738 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603185    6738 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603188    6738 flags.go:64] FLAG: --kube-reserved=""
Oct 30 06:57:19.611904 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603191    6738 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603194    6738 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603197    6738 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603200    6738 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603203    6738 flags.go:64] FLAG: --lock-file=""
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603206    6738 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603210    6738 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603212    6738 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603218    6738 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603221    6738 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603230    6738 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603233    6738 flags.go:64] FLAG: --logging-format="text"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603236    6738 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603241    6738 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603244    6738 flags.go:64] FLAG: --manifest-url=""
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603247    6738 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603252    6738 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603255    6738 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603259    6738 flags.go:64] FLAG: --max-pods="110"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603263    6738 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603266    6738 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603270    6738 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603273    6738 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603276    6738 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603279    6738 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 06:57:19.612610 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603282    6738 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603304    6738 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603308    6738 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603317    6738 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603320    6738 flags.go:64] FLAG: --pod-cidr=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603324    6738 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603326    6738 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603330    6738 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603333    6738 flags.go:64] FLAG: --port="10250"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603336    6738 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603339    6738 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-004c015cc3e17b04a"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603342    6738 flags.go:64] FLAG: --qos-reserved=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603345    6738 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603348    6738 flags.go:64] FLAG: --register-node="true"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603352    6738 flags.go:64] FLAG: --register-with-taints=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603356    6738 flags.go:64] FLAG: --registry-burst="10"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603359    6738 flags.go:64] FLAG: --registry-qps="5"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603362    6738 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603372    6738 flags.go:64] FLAG: --reserved-memory=""
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603377    6738 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603386    6738 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603390    6738 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603393    6738 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603396    6738 flags.go:64] FLAG: --runonce="false"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603399    6738 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603402    6738 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603405    6738 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 06:57:19.613310 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603408    6738 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603412    6738 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603415    6738 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603419    6738 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603422    6738 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603425    6738 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603428    6738 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603431    6738 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603434    6738 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603437    6738 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603440    6738 flags.go:64] FLAG: --system-cgroups=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603443    6738 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603449    6738 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603452    6738 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603454    6738 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603462    6738 flags.go:64] FLAG: --tls-min-version=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603465    6738 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603468    6738 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603471    6738 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603474    6738 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603477    6738 flags.go:64] FLAG: --v="2"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603482    6738 flags.go:64] FLAG: --version="false"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603486    6738 flags.go:64] FLAG: --vmodule=""
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603490    6738 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.603499    6738 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 06:57:19.613946 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603688    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603691    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603694    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603697    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603700    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603702    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603705    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603708    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603712    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603715    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603717    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603720    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603723    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603726    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603729    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603731    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603734    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603736    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603739    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.614590 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603742    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603745    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603747    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603750    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603752    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603755    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603759    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603762    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603765    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603768    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603770    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603773    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603775    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603785    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603788    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603794    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603797    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603799    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603802    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603805    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.615083 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603807    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603811    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603813    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603816    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603818    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603821    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603823    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603826    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603828    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603831    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603833    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603836    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603839    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603841    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603843    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603846    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603848    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603851    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603855    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.615646 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603858    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603861    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603864    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603867    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603869    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603872    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603875    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603883    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603886    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603888    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603891    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603894    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603897    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603899    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603902    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603904    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603907    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603911    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603914    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.616139 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603917    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603920    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603923    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603925    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603928    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603931    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603933    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603936    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603938    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603941    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603944    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603946    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603951    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603954    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603956    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603959    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603961    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603964    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603967    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603969    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.616848 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603972    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603980    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603984    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603987    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603990    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603993    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603996    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.603998    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604001    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.604029    6738 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604078    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604082    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604086    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604089    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604092    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604094    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.617747 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604097    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604100    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604102    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604105    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604107    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604110    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604113    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604115    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604119    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604122    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604124    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604127    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604129    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604132    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604134    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604137    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604140    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604142    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604150    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604153    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.618511 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604156    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604158    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604161    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604163    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604166    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604170    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604172    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604175    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604178    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604180    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604183    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604186    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604188    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604191    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604194    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604196    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604199    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604201    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604204    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.619214 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604207    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604210    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604213    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604216    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604218    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604221    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604224    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604226    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604229    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604231    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604234    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604237    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604246    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604249    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604251    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604254    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604258    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604262    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604266    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.619772 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604269    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604272    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604274    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604277    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604280    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604282    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604285    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604287    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604303    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604306    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604309    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604318    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604320    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604323    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604327    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604330    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604333    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604335    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604338    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604341    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.620226 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604343    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604347    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604351    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604354    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604356    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604359    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604367    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604370    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604373    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604375    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604378    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604382    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604384    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604387    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604390    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604392    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604395    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604397    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604400    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604402    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.620722 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604406    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.604409    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.610081    6738 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.610098    6738 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.610122    6738 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.610127    6738 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610194    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610199    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610204    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610208    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610211    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610214    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610217    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610219    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610223    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610225    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610228    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610231    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610234    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.621240 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610236    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610239    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610243    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610246    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610249    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610252    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610255    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610258    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610261    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610264    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610267    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610270    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610272    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610275    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610277    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610280    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610282    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610285    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610288    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610308    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.621755 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610313    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610327    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610329    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610332    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610335    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610338    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610340    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610343    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610346    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610349    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610351    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610354    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610356    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610359    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610362    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610365    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610368    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610371    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610373    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.622333 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610376    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610379    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610381    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610384    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610387    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610389    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610392    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610396    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610398    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610401    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610403    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610406    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610409    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610411    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610414    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610416    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610419    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610422    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610425    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.622802 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610427    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610430    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610432    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610435    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610438    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610440    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610442    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610445    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610454    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610457    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610459    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610462    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610465    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610469    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610473    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610476    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610479    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610482    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610485    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.623262 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610488    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610490    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610493    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610495    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610498    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610501    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610503    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610506    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610509    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610511    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610514    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610517    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610520    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610522    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610525    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610527    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.623812 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.610555    6738 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610708    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610713    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610716    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610719    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610722    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610730    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610734    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610736    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610739    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610743    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610747    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610751    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610754    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610756    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610759    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610762    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610764    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610767    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610770    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.624224 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610772    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610775    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610778    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610780    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610783    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610785    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610788    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610790    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610795    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610798    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610801    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610803    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610806    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610809    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610811    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610814    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610817    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610819    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610822    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610826    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.624736 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610828    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610831    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610834    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610836    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610839    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610842    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610844    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610847    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610849    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610852    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610854    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610857    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610859    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610862    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610864    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610867    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610870    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610872    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610875    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.625496 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610877    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610884    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610888    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610891    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610894    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610896    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610899    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610902    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610905    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610908    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610910    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610913    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610915    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610918    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610921    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610924    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610926    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610929    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610931    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.626130 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610934    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610937    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610940    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610942    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610945    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610948    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610950    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610953    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610956    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610958    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610961    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610963    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610966    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610969    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610973    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610975    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610978    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610980    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610983    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.626894 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610985    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610988    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610991    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610993    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610996    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.610999    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611001    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611004    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611008    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611011    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.611034    6738 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611066    6738 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611069    6738 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611073    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611076    6738 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611078    6738 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:19.627376 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611081    6738 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611084    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611086    6738 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611089    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611092    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611094    6738 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611097    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611100    6738 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611102    6738 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611105    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611108    6738 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611110    6738 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611114    6738 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611117    6738 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611120    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611123    6738 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611125    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611128    6738 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611131    6738 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:19.627763 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611134    6738 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611136    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611139    6738 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611142    6738 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611145    6738 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611150    6738 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611153    6738 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611156    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611159    6738 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611161    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611164    6738 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611166    6738 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611169    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611171    6738 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611174    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611177    6738 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611179    6738 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611182    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611185    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:19.628253 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611187    6738 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611190    6738 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611193    6738 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611195    6738 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611198    6738 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611201    6738 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611205    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611207    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611210    6738 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611213    6738 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611215    6738 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611218    6738 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611222    6738 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611226    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611229    6738 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611232    6738 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611234    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611237    6738 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611242    6738 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611245    6738 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:19.628745 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611248    6738 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611251    6738 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611254    6738 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611256    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611259    6738 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611261    6738 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611264    6738 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611266    6738 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611269    6738 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611271    6738 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611274    6738 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611277    6738 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611280    6738 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611283    6738 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611286    6738 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611289    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611307    6738 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611310    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611314    6738 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611317    6738 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:19.629233 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611320    6738 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611323    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611325    6738 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611328    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611330    6738 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611333    6738 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611336    6738 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611338    6738 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611341    6738 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611343    6738 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611347    6738 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611350    6738 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611353    6738 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611361    6738 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611364    6738 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611366    6738 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611369    6738 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611371    6738 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611374    6738 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611377    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:19.629786 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611379    6738 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611382    6738 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:19.611387    6738 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.611448    6738 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612431    6738 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612491    6738 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612671    6738 server.go:1014] "Starting client certificate rotation"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612862    6738 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612910    6738 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:07 +0000 UTC" deadline="2028-03-31 00:40:19.841112268 +0000 UTC"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.612963    6738 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12425h43m0.228151484s"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.613163    6738 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.614253    6738 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.615849    6738 log.go:25] "Validated CRI v1 runtime API"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.618465    6738 log.go:25] "Validated CRI v1 image API"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.619142    6738 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 06:57:19.630263 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.621557    6738 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81221195-5e9f-4a8d-af0a-8aa778b15f34:/dev/nvme0n1p4 a5bc53f9-abac-4222-81c4-0bf3cb698942:/dev/nvme0n1p3]
Oct 30 06:57:19.630672 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.621579    6738 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/188aa4ebac4aeea496b6b15daa92541f2fcee0ad44d46122bee97e3e42c717f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/188aa4ebac4aeea496b6b15daa92541f2fcee0ad44d46122bee97e3e42c717f5/userdata/shm major:0 minor:424 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1/userdata/shm major:0 minor:415 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25211ed215be44f2f09fa6aa18d9c10188ab5f4748f3f2d69837f6f1a450885a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25211ed215be44f2f09fa6aa18d9c10188ab5f4748f3f2d69837f6f1a450885a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcf73f3bf23b6fac430bf4052608737530ef132e2f82fe992fbbcbbf17dc7860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcf73f3bf23b6fac430bf4052608737530ef132e2f82fe992fbbcbbf17dc7860/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34/userdata/shm major:0 minor:422 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01063b0f-0dd4-40ac-abe3-edda6dcda17c/volumes/kubernetes.io~projected/kube-api-access-m4ctg:{mountpoint:/var/lib/kubelet/pods/01063b0f-0dd4-40ac-abe3-edda6dcda17c/volumes/kubernetes.io~projected/kube-api-access-m4ctg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~projected/kube-api-access-bctkj:{mountpoint:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~projected/kube-api-access-bctkj major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~projected/kube-api-access-tj9fj:{mountpoint:/var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~projected/kube-api-access-tj9fj major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~projected/kube-api-access-c9bfr:{mountpoint:/var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~projected/kube-api-access-c9bfr major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19df0167-7313-4ba8-a63f-99ce3cd97697/volumes/kubernetes.io~projected/kube-api-access-9kd62:{mountpoint:/var/lib/kubelet/pods/19df0167-7313-4ba8-a63f-99ce3cd97697/volumes/kubernetes.io~projected/kube-api-access-9kd62 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~projected/kube-api-access-2g4xp:{mountpoint:/var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~projected/kube-api-access-2g4xp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f128ed7-b717-4188-84f8-0ea3f320dee4/volumes/kubernetes.io~projected/kube-api-access-c7wv7:{mountpoint:/var/lib/kubelet/pods/2f128ed7-b717-4188-84f8-0ea3f320dee4/volumes/kubernetes.io~projected/kube-api-access-c7wv7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~projected/kube-api-access-dwjh4:{mountpoint:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~projected/kube-api-access-dwjh4 major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~projected/kube-api-access-ch8ls:{mountpoint:/var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~projected/kube-api-access-ch8ls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~projected/kube-api-access-5nj72:{mountpoint:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~projected/kube-api-access-5nj72 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5eb0f2a4-b0bb-4259-9f93-0e2331129305/volumes/kubernetes.io~projected/kube-api-access-xf9nd:{mountpoint:/var/lib/kubelet/pods/5eb0f2a4-b0bb-4259-9f93-0e2331129305/volumes/kubernetes.io~projected/kube-api-access-xf9nd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6528c489-7d36-4881-a608-11ecdd2bf3f3/volumes/kubernetes.io~projected/kube-api-access-j87gk:{mountpoint:/var/lib/kubelet/pods/6528c489-7d36-4881-a608-11ecdd2bf3f3/volumes/kubernetes.io~projected/kube-api-access-j87gk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~projected/kube-api-access-2fgc4:{mountpoint:/var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~projected/kube-api-access-2fgc4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992e92c9-6236-4e46-90bb-08f2a0b89a17/volumes/kubernetes.io~projected/kube-api-access-bk2nn:{mountpoint:/var/lib/kubelet/pods/992e92c9-6236-4e46-90bb-08f2a0b89a17/volumes/kubernetes.io~projected/kube-api-access-bk2nn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2098dac-6268-493b-b74d-d93639ac7e8d/volumes/kubernetes.io~projected/kube-api-access-dxhtn:{mountpoint:/var/lib/kubelet/pods/c2098dac-6268-493b-b74d-d93639ac7e8d/volumes/kubernetes.io~projected/kube-api-access-dxhtn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~projected/kube-api-access-8hrnm:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~projected/kube-api-access-8hrnm major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~projected/kube-api-access-v78hq:{mountpoint:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~projected/kube-api-access-v78hq major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~projected/kube-api-access-bmqjh:{mountpoint:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~projected/kube-api-access-bmqjh major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/alertmanager/9 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/config-reloader/6 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/kube-api-access-b276w:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/kube-api-access-b276w major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/tls-assets major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/config-volume major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/web-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~projected/kube-api-access-n5cxl:{mountpoint:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~projected/kube-api-access-n5cxl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5506e4ecb81b8f7b4f6b184152a35cf39896f63c1b62908ff10d747cd27db5cb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3e1f5042c942ca1452ea7ce63b7d1701a4756af660cf9a6981d72dc816152f6c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ffec2dab0f40627613e426efa665a3939331ea55adf9ca2c5476fe55961a7df3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ad9cb36cdca7f545f67e476a37bc4949eb211b07219cc280d77520b909141323/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/140763f8ba09540a6d3c0230e895b6bffa710c4f315e36d886bca3e7982e300d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c0dcd65be6859ea317270f40875f476636c4112442b5545e4e0f8c2707538133/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/77d022af74f74f82b5177b43a00c8e5ea42495bbe1a48b63c4bddc57daeb39a9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8dff3a0adc1dd4ad1bb559daf0dc81b4892e98ec9de2b3486850503026714172/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c2e2be481bb7a49443ca71027ed41331e185958643fecd72554b39fbffb61f27/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f35670a22800813c7b330a3a29dac9597f6e4e75951357c859eb33ac34f32078/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/0fc4133ca6f7886e603dd284ab38d2243bf31aec84ff35cb20b0309d412d0b24/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/d1cca09e48e438d17c3f3d97fb5929ace86be7253d2c5ff9896c7bd830d67467/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c229626e08f13f6884f6af5f9c15f4c694b71842895500e0ed48c2bbb56de13c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6d35b3f02ff39e3c6f0bf61864d9a325145e0f51873fe3a27a45680f8c38bc31/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c591b2ae2c35b764424fa1b26e893f235dc9002108df01d458164e6bff7fb59b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c8a3d1930999afe6e20b6e2fb5e1e4a6c317d0b2c58d9f6e3761f637c333420c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a333d45bf05f58fbac41bb864d0e4b049dec7a8b89675c5277837179099a7408/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b53849892674fe7766c8f2e2c481622914715a15873bd59d9d8b4b826c487a4d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/39222e2d5419cd76501e15911c74e0ad1632fb991760362e8ffb52b18231bd07/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c4693be75ba39520c7bbac33dac4bdcc3e2157acf133c9a56f6f7a662883c59d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9b3feb61d3ea4f55555eb6830f515ff1dcf926ac1e6c3d5a962ef12c8a6bb621/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3150bba8a90b72c41640daeadd464504673cbf4392f2044f7a1cd88e65b72a06/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4e5eda62e5859b2d57eba957be44816fa63ba9569e9a65b64588e4af5ffc87c1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/77924b334c9cb9ee0dc8b16f61f5efb3c9e6b7c2bcee0fa6683f1963a4adcc0e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5f7c5be31689fdf44e4bedbf22bab011ef1a8ccf04b0caf1805ca4930d83e5e2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b0d60f9e653399ff188c7f1b80a685660591923556ac6d271ce2877fcdc7c289/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/da1a3a481283f2197a1abff5f5ad9f5534e6bad9176665e87c935123f13d3f36/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/070b1733aa1231732f8a811c575722bcf1a4cb87c0bbc73018672fba01003252/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/239010bb22b119af665885b01b808ced6da12109eb431a173e6d13617ef0cbdb/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d759b33f4374c1f70784bda1a59c09c64121f4f2072c0e0afb967163ac629a0d/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/8bf8737c16e57eb81c33386039bf46ba4de0c68009287b31351db97807925159/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/9f2d1d983c0b11f33fc2783b97a9f9bd76dc701b67e66e33058545081e2450fc/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/e431df453f9f1c235413eb69495f35dbe9af79288cfb03faaf31948f9b9a11f2/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f268bc0489a08e3bc1dc81b825543cb25cc74d81f0be268d21b3c959b40c3b6d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a0b71f7e18ec2b0b5d0805bcc3813c160c95b908f5b479a3f1497bdf0d8a8c58/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/ac7a160cdabd435710fc2e861787f9b3fb2c324556e577f0d55ea91c64be1312/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/47bb17ee5122b9cc36bba874ad2ebc0441012a94b8d00dc935f490591017a5a1/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/d7fa6da95e0849f5fadca9a6768c69b9c5159dd9b7ff6fccf1375ee7ae936a0d/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a03e3bffcfc70d2507346f59236858c93dd29ed10d1866d9f5f9632b6b6b89ba/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/9de68bdbb3f0b222c253b5f4ae9274d391bbaad75f0a23720d05eb13d062ca73/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/0d8a4f2a7311491316a899daf35c8be1fc9957d183c533dd8a902a6555e9d704/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/54f809f072648c7e3ef67d749264a8eb5577f40bbb995396a53b0f352e9b9751/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/0a4f7754d63c24f61338f8e7b8660dbca48e7b8ce577ac70a85165b8f43e129e/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/86a9daf30330512ab853ae317a20421e80c004c232e4acde7fc2b7565ca97a98/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/9250d3d7a370488396db91915bb70fd358fbcae3eb8e69a454fd5e68bd3e4ef6/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/1437fa520ced2399c7eb78ade1799896b30b2b4325221d3f6e36b888ef859b69/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/67af288f626797eded8fa140e1c10cd105dca38f4107275d32b5607936a73f34/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7bc445af2fe5f5eae31259a178428afe5ce286042931e253838a1e3024394810/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/a0d6af675bc0dc5b49c897c7c8ec5940de0420ab557ba035d339cf37cc4df1fe/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/94309ee61f4b6f450b7532a1d36cd3c30c3eeac0f4879832fc2fab6e85dc8d98/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/3a866d8a4e94bc2236f119cc3500bf13799737128dd41f9529cb9bcc3eb8c79f/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/6f0a2d9a5c1418b0b3b7f7822e000f59a2a94cdb1587114370e6321375ab40ab/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/3d359eb469fd91a8c75eddd1a87b6b39818f03b1953d7c03417e563ac1ae13bb/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/8756a70ddfa919224fab679c1fbbd9c092b5ffba34e423dca9ef1308aad31d1d/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/d7f90a0e050ce5e88e69ee38ff3094789c283a31d7b4b0045d6398696d0fc146/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/2dcaf1d7bf7f832150fc6cc38c6b7697e56c39b2cd1b8691a75d68a71457989d/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/9937ee68ac2e4b1218d2e8128e916b1604a8bf6a9370f4f005468d5d6ce6282c/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/6d85d9d02e32529b4d6ef52ee45d49d58b8f76c135580f298130eeeecb4f2b77/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/6588deea11ee5992c5778f01aa7baf5eb9e6deb620ac73cb2085e830cf86f330/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/a964ea2dab62fe397264c2e133bfb03b5a33001ac77ff311291880ae17da0e1f/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/44fc1608ce5f6f32bffa69d462cab5a4ee877c770f143ebe994a0c3e5ccf9c92/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/63e492c1518d58f3ec6c8187cf028af7ed706c84959a304da23a5204b2e07907/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/6df305c4546f5f7ca6a68fc2429f3eda3a34e44b55994e96792f0c6392e74409/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/d7521a94bc6e2767eae0a9ebebec5a77b81048ef5700775b47c5980b15722ab3/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/ff78a739f999848374e6d065556f5f2a2a408bcfe4f9a4ff1f93e46479fd21e1/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/bb4f77692b47ef491ba4e0930f1234f57fa986eded369fa2b65e31d22201e8d4/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/c413285f54389f9c7b80f36e92be2c87c878b17326e8be62c9731128c1058429/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d0910d2c9347bc81690e15912cb1556a790f14646ea9d61a36ab3af00047f26e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/29480c3870edc8a10db1b08cdf4e7423dea644db17fb1a1cfa6ae06cb57a604b/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/207d3691d585ee5ed95754be2a029626ffd1acc6e14c01354b814e8c5729bb3e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/aac42d741a50c99e3635597059c9d01d339c5f22255dec61a0afae213243254e/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/6dcc8ff740306910adf35a64e50d46deee8f4c9ff62a9f184a143bb5a043d0d4/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/cd7e30a028c01a62b9e8b48b2b4bc0d7662ed4759b1cd63a145aa38547a37231/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/178afddf5d8d59ea2fce172e638106e5d8ffe347208079fa0f18dc0900b7759e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/675b9b97626f0830cc80e417f699b8f07c49e77f471dd3e05248d2610ed4c6a9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d5dbc7e790e7a0f461bc361e2cfeff56a8184cd799aa6b3257bd7fc79d7edeae/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 06:57:19.632401 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.631982    6738 manager.go:217] Machine: {Timestamp:2026-10-30 06:57:19.631477909 +0000 UTC m=+0.064241793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100001 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b44a83e7fb9c4814e32cb1c3d1920 SystemUUID:ec2b44a8-3e7f-b9c4-814e-32cb1c3d1920 BootID:eb5d9a8d-af65-478b-8ff0-8cc75654e403 Filesystems:[{Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a646d5d913566235e3a7bac98ff5ec9d1297bf583d5e912be18dd46480b1bc0b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~projected/kube-api-access-ch8ls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~projected/kube-api-access-8hrnm DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:413 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~projected/kube-api-access-n5cxl DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~projected/kube-api-access-v78hq DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcf73f3bf23b6fac430bf4052608737530ef132e2f82fe992fbbcbbf17dc7860/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992e92c9-6236-4e46-90bb-08f2a0b89a17/volumes/kubernetes.io~projected/kube-api-access-bk2nn DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~projected/kube-api-access-5nj72 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~projected/kube-api-access-2g4xp DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eeb51e17dfc628f42c8a410dd1c273cc64f317f70cdda160babfe952fc0f80c5/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfb7ffeefc47a584ea51e6ccf9680008a59e1aa72a952771ab8171f7398a3c34/userdata/shm DeviceMajor:0 DeviceMinor:422 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~projected/kube-api-access-bmqjh DeviceMajor:0 DeviceMinor:256 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~projected/kube-api-access-bctkj DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0947f45dfedf1fa632f1dd39400db940a3d481dc63fd7cc1784440859edd2d1d/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/012d3e359a45cfeb05e2d9877d251a724445a545d9405ea9770a97070fcf9569/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/334abc08cea29c09336c9d318b6846d3c729d34353492a23773b5b12fa2f4ee9/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25211ed215be44f2f09fa6aa18d9c10188ab5f4748f3f2d69837f6f1a450885a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f128ed7-b717-4188-84f8-0ea3f320dee4/volumes/kubernetes.io~projected/kube-api-access-c7wv7 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/358670d6-0bd0-4c3b-81cb-beafe2872176/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f02f2149-de58-41d0-892a-77532f24a26b/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:409 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d3eea62-15e2-488a-a427-abb5e81d034b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:419 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8094e4c0a73b710ba57f42e1ede6deb6189b11430faa5470495b8487f8394cd7/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8c17490477d80a0daf71ce2b267960683679c6c1d8dd9cae008f125825265ca/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91441c1fcbc166b08577c8bd791ab54e78b591b02e421b37c1e36c2bee985419/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85cf48dbd4d834c2b6165706ed30fbb71e0a0d02a02de1d291d704b457fced68/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6528c489-7d36-4881-a608-11ecdd2bf3f3/volumes/kubernetes.io~projected/kube-api-access-j87gk DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19ee6f98528211f8f72342179ce55dee05b0ed41a599715f9b8d5016f1228a13/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes/kubernetes.io~projected/kube-api-access-dwjh4 DeviceMajor:0 DeviceMinor:414 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19df0167-7313-4ba8-a63f-99ce3cd97697/volumes/kubernetes.io~projected/kube-api-access-9kd62 DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5eb0f2a4-b0bb-4259-9f93-0e2331129305/volumes/kubernetes.io~projected/kube-api-access-xf9nd DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aa33e7702a0870d4d7da7ce9acb649c3a534e7be40dd05d02b4ffc8533b4e4e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da285341224f2a140fec886f41c5548cc08fad468bf6f27881e85176444ff8d0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f5f20e3fd22c49b17d32c159a4363268e462773dcd851d7d2ccfa7ad8425ddc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10f67d3c-20ef-48ee-b6a5-c33e841afd3d/volumes/kubernetes.io~projected/kube-api-access-tj9fj DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2098dac-6268-493b-b74d-d93639ac7e8d/volumes/kubernetes.io~projected/kube-api-access-dxhtn DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de524ef776dca54332f0f1b60a44b9ba8989ee487d7d5d5dc4b13ca8eb8e64ba/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53178620f408520b80bc61dda206cd90e2a22591226ae2382087304e6f76d3a9/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17f2a963-5150-4193-826b-bfd5a336ce5b/volumes/kubernetes.io~projected/kube-api-access-c9bfr DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7de252f7-3dda-4a15-abb2-a69f2b22fb7e/volumes/kubernetes.io~projected/kube-api-access-2fgc4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6830ae7-bf50-490b-bbf8-f5d32835edfb/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~projected/kube-api-access-b276w DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f2a8f9c-d148-4051-855b-79b8da1b57dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1/userdata/shm DeviceMajor:0 DeviceMinor:415 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/188aa4ebac4aeea496b6b15daa92541f2fcee0ad44d46122bee97e3e42c717f5/userdata/shm DeviceMajor:0 DeviceMinor:424 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb43b53-7a8a-44d5-a802-2eb2691e2758/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01063b0f-0dd4-40ac-abe3-edda6dcda17c/volumes/kubernetes.io~projected/kube-api-access-m4ctg DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42f96032-aac6-45a7-b77b-e7f43f33afbc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:012d3e359a45cfe MacAddress:46:c2:4f:4a:b0:14 Speed:10000 Mtu:8901} {Name:0947f45dfedf1fa MacAddress:82:e5:4e:6b:f9:f1 Speed:10000 Mtu:8901} {Name:188aa4ebac4aeea MacAddress:de:b1:86:e4:df:c1 Speed:10000 Mtu:8901} {Name:1d4e2ed1e0d2b9a MacAddress:b6:b7:c9:ea:e3:cc Speed:10000 Mtu:8901} {Name:334abc08cea29c0 MacAddress:1e:04:3a:52:d3:7e Speed:10000 Mtu:8901} {Name:53178620f408520 MacAddress:0a:b3:6e:50:af:6d Speed:10000 Mtu:8901} {Name:6941cfde893bd7a MacAddress:ee:31:3e:69:05:36 Speed:10000 Mtu:8901} {Name:7f5f20e3fd22c49 MacAddress:fa:c0:e2:53:64:c2 Speed:10000 Mtu:8901} {Name:85cf48dbd4d834c MacAddress:4e:b1:55:b3:6a:47 Speed:10000 Mtu:8901} {Name:91441c1fcbc166b MacAddress:12:b4:e4:bc:be:39 Speed:10000 Mtu:8901} {Name:bcf73f3bf23b6fa MacAddress:56:ab:4a:13:81:34 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:12:00:af Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:60:3a:41:57:b0 Speed:0 Mtu:8901} {Name:cfb7ffeefc47a58 MacAddress:fa:6c:5c:b1:3c:04 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:12:00:af Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f6:38:f0:c8:95:c4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:c3:a4:35:17:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 06:57:19.632701 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.632405    6738 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 06:57:19.632701 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.632459    6738 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 06:57:19.632868 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.632839    6738 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 06:57:19.633011 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.632870    6738 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 06:57:19.633052 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633015    6738 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 06:57:19.633052 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633023    6738 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 06:57:19.633052 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633038    6738 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:57:19.633131 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633052    6738 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:57:19.633177 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633169    6738 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 06:57:19.633411 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633400    6738 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 06:57:19.633525 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633517    6738 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 06:57:19.633563 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633555    6738 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 06:57:19.633592 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633567    6738 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 06:57:19.633592 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633585    6738 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 06:57:19.633659 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633594    6738 kubelet.go:405] "Adding apiserver pod source"
Oct 30 06:57:19.633659 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.633604    6738 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 06:57:19.634548 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.634531    6738 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 06:57:19.634548 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.634544    6738 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:57:19.635127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.635108    6738 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 06:57:19.638063 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638047    6738 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 06:57:19.638063 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638060    6738 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:57:19.638184 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638154    6738 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 06:57:19.638367 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638355    6738 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 06:57:19.638406 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638385    6738 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 06:57:19.638485 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638476    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638489    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638495    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638501    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638506    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638513    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 06:57:19.638515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638518    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638524    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638531    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638537    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638545    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638555    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638581    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 06:57:19.638735 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.638588    6738 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 06:57:19.642044 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.642032    6738 server.go:1263] "Started kubelet"
Oct 30 06:57:19.642141 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.642109    6738 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 06:57:19.642207 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.642137    6738 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 06:57:19.642272 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.642233    6738 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 06:57:19.643261 ip-10-0-139-124 systemd[1]: Started Kubernetes Kubelet.
Oct 30 06:57:19.645783 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.643554    6738 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 06:57:19.646273 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.646258    6738 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.647930    6738 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.647960    6738 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.648010    6738 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.648019    6738 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:19.648045    6738 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-124.ec2.internal\" not found"
Oct 30 06:57:19.648284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.648100    6738 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650451    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6528c489-7d36-4881-a608-11ecdd2bf3f3" volumeName="kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650499    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650512    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650524    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650535    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650547    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650564    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358670d6-0bd0-4c3b-81cb-beafe2872176" volumeName="kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650575    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2098dac-6268-493b-b74d-d93639ac7e8d" volumeName="kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650589    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650600    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650612    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650624    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650636    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650647    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f128ed7-b717-4188-84f8-0ea3f320dee4" volumeName="kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650671    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f96032-aac6-45a7-b77b-e7f43f33afbc" volumeName="kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650683    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650693    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02f2149-de58-41d0-892a-77532f24a26b" volumeName="kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650704    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb43b53-7a8a-44d5-a802-2eb2691e2758" volumeName="kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650716    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063b0f-0dd4-40ac-abe3-edda6dcda17c" volumeName="kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650729    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063b0f-0dd4-40ac-abe3-edda6dcda17c" volumeName="kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650740    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f2a963-5150-4193-826b-bfd5a336ce5b" volumeName="kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650750    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358670d6-0bd0-4c3b-81cb-beafe2872176" volumeName="kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650770    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358670d6-0bd0-4c3b-81cb-beafe2872176" volumeName="kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650787    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de252f7-3dda-4a15-abb2-a69f2b22fb7e" volumeName="kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650799    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de252f7-3dda-4a15-abb2-a69f2b22fb7e" volumeName="kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650810    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992e92c9-6236-4e46-90bb-08f2a0b89a17" volumeName="kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650822    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650834    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6528c489-7d36-4881-a608-11ecdd2bf3f3" volumeName="kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650857    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650870    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650881    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650892    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb43b53-7a8a-44d5-a802-2eb2691e2758" volumeName="kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650904    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f96032-aac6-45a7-b77b-e7f43f33afbc" volumeName="kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650915    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c36d37c-2e1a-41aa-9f1a-bf67e50331c9" volumeName="kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650926    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650938    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063b0f-0dd4-40ac-abe3-edda6dcda17c" volumeName="kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650949    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650961    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02f2149-de58-41d0-892a-77532f24a26b" volumeName="kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650972    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650983    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb43b53-7a8a-44d5-a802-2eb2691e2758" volumeName="kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.650994    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d3eea62-15e2-488a-a427-abb5e81d034b" volumeName="kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651007    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c36d37c-2e1a-41aa-9f1a-bf67e50331c9" volumeName="kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651017    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651028    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651040    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02f2149-de58-41d0-892a-77532f24a26b" volumeName="kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651051    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651068    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651081    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651092    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651103    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651115    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651130    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2098dac-6268-493b-b74d-d93639ac7e8d" volumeName="kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651142    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02f2149-de58-41d0-892a-77532f24a26b" volumeName="kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651159    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb43b53-7a8a-44d5-a802-2eb2691e2758" volumeName="kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651171    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="358670d6-0bd0-4c3b-81cb-beafe2872176" volumeName="kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651183    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992e92c9-6236-4e46-90bb-08f2a0b89a17" volumeName="kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651202    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651214    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb43b53-7a8a-44d5-a802-2eb2691e2758" volumeName="kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651225    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651238    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f67d3c-20ef-48ee-b6a5-c33e841afd3d" volumeName="kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651250    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651261    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651272    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651283    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651309    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651321    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f67d3c-20ef-48ee-b6a5-c33e841afd3d" volumeName="kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651333    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de252f7-3dda-4a15-abb2-a69f2b22fb7e" volumeName="kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651344    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992e92c9-6236-4e46-90bb-08f2a0b89a17" volumeName="kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651355    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91" volumeName="kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651365    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651378    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f96032-aac6-45a7-b77b-e7f43f33afbc" volumeName="kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651389    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de252f7-3dda-4a15-abb2-a69f2b22fb7e" volumeName="kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651400    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f2a963-5150-4193-826b-bfd5a336ce5b" volumeName="kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651412    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063b0f-0dd4-40ac-abe3-edda6dcda17c" volumeName="kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651423    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651439    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19df0167-7313-4ba8-a63f-99ce3cd97697" volumeName="kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651450    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f128ed7-b717-4188-84f8-0ea3f320dee4" volumeName="kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651461    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eb0f2a4-b0bb-4259-9f93-0e2331129305" volumeName="kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651473    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2ab3bbf-b449-4b3c-b048-5180eb3b2e37" volumeName="kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651484    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651496    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651507    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6830ae7-bf50-490b-bbf8-f5d32835edfb" volumeName="kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651517    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa866eab-cad8-49ed-9c13-52a4f33605a8" volumeName="kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651530    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f2a8f9c-d148-4051-855b-79b8da1b57dc" volumeName="kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651541    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d3eea62-15e2-488a-a427-abb5e81d034b" volumeName="kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651551    6738 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" volumeName="kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert" seLinuxMountContext=""
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651563    6738 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651570    6738 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.651982    6738 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:09 +0000 UTC" deadline="2028-05-05 21:27:04.783719722 +0000 UTC"
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.652005    6738 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13286h29m45.131718185s"
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.654696    6738 factory.go:55] Registering systemd factory
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.654715    6738 factory.go:223] Registration of the systemd container factory successfully
Oct 30 06:57:19.654988 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655070    6738 factory.go:153] Registering CRI-O factory
Oct 30 06:57:19.659560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655092    6738 factory.go:223] Registration of the crio container factory successfully
Oct 30 06:57:19.659560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655158    6738 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 06:57:19.659560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655181    6738 factory.go:103] Registering Raw factory
Oct 30 06:57:19.659560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655195    6738 manager.go:1199] Started watching for new ooms in manager
Oct 30 06:57:19.659560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.655577    6738 manager.go:319] Starting recovery of all containers
Oct 30 06:57:19.665785 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.665746    6738 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 06:57:19.681695 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.681659    6738 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 06:57:19.681695 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.681690    6738 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 06:57:19.681894 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.681710    6738 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 06:57:19.681894 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:19.681767    6738 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 06:57:19.699628 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.699597    6738 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:19.699863 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.699800    6738 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:19.699963 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.699905    6738 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705270    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="9ae7e7babe1735fc60d0dfb10af6a241da92813289de924adac26b1286fe9c3d" exitCode=0
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705310    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="dc75dc8d6fefd5446ca1f1bfa2bf3d4cec62c44b908fc0fedf3cdb99b8790591" exitCode=0
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705319    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="cc7f6e1a5ca3c2ddadb4b4d335603951320427143f9f37579aa25de3fd3210f9" exitCode=0
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705326    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="0af955e3058d7d5eb86df2c0466ad1c5d5a3a205c8082a48f7441c85971442c1" exitCode=0
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705333    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="16c67563823f58d47ee989688735dbdb0219f6ce0356ecabf25db0e6bf86961f" exitCode=0
Oct 30 06:57:19.705346 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.705341    6738 generic.go:356] "Generic (PLEG): container finished" podID="01063b0f-0dd4-40ac-abe3-edda6dcda17c" containerID="276f0323c4c47c93d90b15f7ad4372225428c8d2d8f1ba60cad2a6be865253bf" exitCode=0
Oct 30 06:57:19.712538 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.712514    6738 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:19.717628 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.717593    6738 generic.go:356] "Generic (PLEG): container finished" podID="c8de38f9da3fc95cb8d0657cd8e6c898" containerID="4abded307aed34758d8f9e59db4c51f41ebc9b312a6b18d21cb560b85457d22d" exitCode=0
Oct 30 06:57:19.723970 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.723942    6738 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:19.725931 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.725908    6738 generic.go:356] "Generic (PLEG): container finished" podID="f6830ae7-bf50-490b-bbf8-f5d32835edfb" containerID="cfd7637b833ee560b15dd59911a7df0d6e991b74f515369231613ce5d2fe443b" exitCode=0
Oct 30 06:57:19.734330 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.734284    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a" exitCode=0
Oct 30 06:57:19.766672 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.766645    6738 manager.go:324] Recovery completed
Oct 30 06:57:19.782344 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:19.782254    6738 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 06:57:19.789681 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789655    6738 cpu_manager.go:226] "Starting" policy="none"
Oct 30 06:57:19.789681 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789674    6738 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 06:57:19.789837 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789700    6738 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 06:57:19.789875 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789858    6738 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 06:57:19.789922 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789877    6738 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 06:57:19.789922 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789903    6738 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 06:57:19.789922 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789912    6738 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 06:57:19.789922 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789920    6738 policy_none.go:50] "Start"
Oct 30 06:57:19.790023 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789935    6738 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 06:57:19.790023 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.789950    6738 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 06:57:19.790096 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.790086    6738 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 06:57:19.790126 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.790097    6738 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 06:57:19.790126 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.790103    6738 policy_none.go:44] "Start"
Oct 30 06:57:19.795809 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.795757    6738 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 06:57:19.798756 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.798733    6738 manager.go:341] "Starting Device Plugin manager"
Oct 30 06:57:19.799327 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:19.798773    6738 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 06:57:19.799327 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.799327    6738 server.go:87] "Starting device plugin registration server"
Oct 30 06:57:19.799603 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.799591    6738 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 06:57:19.799675 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.799601    6738 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 06:57:19.799727 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.799689    6738 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 06:57:19.800407 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.800368    6738 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 06:57:19.800498 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.800418    6738 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 06:57:19.800498 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.800425    6738 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 06:57:19.800586 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.800479    6738 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:57:19.800402395Z","UUID":"564e4bc1-1be1-4fc8-96f2-9f24e3415fb6","Handler":null,"Name":"","Endpoint":""}
Oct 30 06:57:19.801592 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:19.801570    6738 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 06:57:19.801864 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.801845    6738 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 06:57:19.801942 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.801878    6738 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 06:57:19.900646 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.900604    6738 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:57:19.903208 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.903176    6738 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:57:19.903365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.903236    6738 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:57:19.903365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.903263    6738 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-124.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:57:19.903489 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.903400    6738 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:57:19.941251 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.941216    6738 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:57:19.941431 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.941336    6738 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-124.ec2.internal"
Oct 30 06:57:19.982759 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.982693    6738 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"]
Oct 30 06:57:19.983288 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983257    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal" event={"ID":"bc1bcd324f5af093ecce999f61f1d18f","Type":"ContainerStarted","Data":"21f599a8893a6f1876b69de122c444cf2f3d08a458e0ccc37d2d86f3494c7363"}
Oct 30 06:57:19.983408 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983392    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerStarted","Data":"9d3ca637e417d3d0e0b1f4d45c754a82c21e6fbadd01afa34057526912b43e65"}
Oct 30 06:57:19.983446 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983417    6738 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:19.983502 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983419    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerDied","Data":"4abded307aed34758d8f9e59db4c51f41ebc9b312a6b18d21cb560b85457d22d"}
Oct 30 06:57:19.983537 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983513    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal" event={"ID":"c8de38f9da3fc95cb8d0657cd8e6c898","Type":"ContainerStarted","Data":"25b45d6dbfac7661a2a7ab61546721bd15ea2d12ed23f3adf53819f09ca873d5"}
Oct 30 06:57:19.983537 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:19.983517    6738 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.009091 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.009060    6738 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:20.009200 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.009118    6738 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.009451 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.009434    6738 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:20.009511 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.009470    6738 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.101954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.101871    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.101954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.101910    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.101954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.101926    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202495 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202459    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202495 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202498    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202639 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202514    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202639 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202570    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202639 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202622    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8de38f9da3fc95cb8d0657cd8e6c898-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" (UID: \"c8de38f9da3fc95cb8d0657cd8e6c898\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.202739 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.202655    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc1bcd324f5af093ecce999f61f1d18f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-124.ec2.internal\" (UID: \"bc1bcd324f5af093ecce999f61f1d18f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.634392 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.634356    6738 apiserver.go:52] "Watching apiserver"
Oct 30 06:57:20.688563 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.688529    6738 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 06:57:20.689197 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.689171    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n75wj","kube-system/konnectivity-agent-djbct","openshift-multus/network-metrics-daemon-vtjnb","openshift-monitoring/node-exporter-vns7r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv","openshift-multus/multus-9mh4r","openshift-network-diagnostics/network-check-target-qdhdb","openshift-dns/node-resolver-qgqkv","openshift-multus/multus-additional-cni-plugins-wp2gn","openshift-ovn-kubernetes/ovnkube-node-6jgfh","openshift-dns/dns-default-wmj4w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk","openshift-monitoring/thanos-querier-86b57b86db-sjlhb","openshift-image-registry/node-ca-nxvzr","kube-system/global-pull-secret-syncer-f2pl2","kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal","openshift-console/console-666d7bbf55-jm86p","openshift-cluster-node-tuning-operator/tuned-gc6hx","openshift-ingress-canary/ingress-canary-7r8w2","openshift-insights/insights-runtime-extractor-rqlss","openshift-network-operator/iptables-alerter-hr547"]
Oct 30 06:57:20.706586 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.706551    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 06:57:20.706938 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.706913    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707072 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707003    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707072 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707009    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707072 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707035    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 06:57:20.707072 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707036    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44sjc\""
Oct 30 06:57:20.707072 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707053    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707078    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707101    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707102    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-th56t\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707114    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707221    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707258    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707275    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707327    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 06:57:20.707334 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707334    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707344    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707345    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707363    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707363    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707326    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707395    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707413    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707414    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707420    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s8tns\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707457    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q4jsx\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707479    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7szgh\""
Oct 30 06:57:20.707645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.707529    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2h44\""
Oct 30 06:57:20.718523 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.718497    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 06:57:20.718652 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.718539    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:57:20.718652 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.718546    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 06:57:20.719138 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719124    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 06:57:20.719247 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719233    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.719523 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719509    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.719888 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719874    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 06:57:20.719994 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719960    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719991    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.719999    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720029    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720131    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720143    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 06:57:20.720160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720153    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720256    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720308    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720351    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720337    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720345    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720336    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:57:20.720520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720421    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 06:57:20.720923 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720909    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vngf\""
Oct 30 06:57:20.720999 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720910    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4pt2f\""
Oct 30 06:57:20.720999 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.720990    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:57:20.721099 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.721009    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6h5pm\""
Oct 30 06:57:20.721099 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.721045    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ldd2l\""
Oct 30 06:57:20.721099 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.721089    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pptc7\""
Oct 30 06:57:20.722171 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722157    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 06:57:20.722370 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722241    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 06:57:20.722370 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722233    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 06:57:20.722370 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722312    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-75crj\""
Oct 30 06:57:20.722501 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722419    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 06:57:20.722501 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722486    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 06:57:20.722577 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722511    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 06:57:20.722652 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722638    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 06:57:20.722839 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722822    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 06:57:20.723022 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722996    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b8vbj\""
Oct 30 06:57:20.723022 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723011    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 06:57:20.723022 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723003    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2n2c2\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.722996    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723145    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723151    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723153    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723168    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723180    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 06:57:20.723186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723184    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fqkebqo6dt9ij\""
Oct 30 06:57:20.723478 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.723248    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 06:57:20.731073 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.731054    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2tccl\""
Oct 30 06:57:20.731160 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.731102    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 06:57:20.732214 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732199    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 06:57:20.732265 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732243    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 06:57:20.732330 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732271    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 06:57:20.732330 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732322    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 06:57:20.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732388    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djnbg\""
Oct 30 06:57:20.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732414    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 06:57:20.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732419    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 06:57:20.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732424    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 06:57:20.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732438    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5k2w\""
Oct 30 06:57:20.732608 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.732478    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dpx9k\""
Oct 30 06:57:20.733587 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.733569    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 06:57:20.749833 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.749807    6738 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.751282 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.751267    6738 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 06:57:20.772415 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.772389    6738 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:20.772510 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.772444    6738 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-124.ec2.internal"
Oct 30 06:57:20.805282 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805253    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.805417 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805287    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2nn\" (UniqueName: \"kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.805417 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805332    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.805536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805507    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrnm\" (UniqueName: \"kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.805586 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805568    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.805633 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805581    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-cni-binary-copy\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.805670 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805641    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.805670 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805662    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.805755 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805681    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.805755 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.805690    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:20.805755 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805724    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bfr\" (UniqueName: \"kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:20.805881 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805790    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.805947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.805917    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:20.806004 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.805954    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.305938519 +0000 UTC m=+1.738702418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:20.806069 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-config-volume\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.806069 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806043    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.806188 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806076    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.806188 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806103    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.806188 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806174    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.806365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806208    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.806365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806226    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-tmp\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.806365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806265    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.806365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806356    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.806724 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806708    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.806781 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806744    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.806781 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.806753    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:20.806874 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806773    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.806874 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.806806    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.306790747 +0000 UTC m=+1.739554647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:20.806874 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806825    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.806874 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806853    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806876    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6528c489-7d36-4881-a608-11ecdd2bf3f3-tmp-dir\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806882    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.806977    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807002    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807010    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807014    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-config\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807044    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.807060 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807051    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807148    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87gk\" (UniqueName: \"kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807184    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807203    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807251    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807275    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807284    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807322    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.807424 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807146    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-original-pull-secret\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807440    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807471    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-tls\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807485    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhtn\" (UniqueName: \"kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807537    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807545    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807540    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807631    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807653    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-cert\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807689    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807736    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807766    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807791    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqjh\" (UniqueName: \"kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807815    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807848    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807872    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807873    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-tuned\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.807935 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.807875    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.807891    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808266    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02f2149-de58-41d0-892a-77532f24a26b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808277    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wv7\" (UniqueName: \"kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.808337    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.308307283 +0000 UTC m=+1.741071155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808369    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9nd\" (UniqueName: \"kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808404    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f96032-aac6-45a7-b77b-e7f43f33afbc-tmp\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808411    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808447    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8ls\" (UniqueName: \"kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808488    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-metrics-client-ca\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808497    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808529    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808558    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808583    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.808606 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808600    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/358670d6-0bd0-4c3b-81cb-beafe2872176-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808746    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808806    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808849    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808877    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/358670d6-0bd0-4c3b-81cb-beafe2872176-data-volume\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808918    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.808981    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809115    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809173    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809196    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a8f9c-d148-4051-855b-79b8da1b57dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.809285 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809223    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809413    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809478    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809511    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809520    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-agent-certs\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.809698    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809753    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c36d37c-2e1a-41aa-9f1a-bf67e50331c9-konnectivity-ca\") pod \"konnectivity-agent-djbct\" (UID: \"7c36d37c-2e1a-41aa-9f1a-bf67e50331c9\") " pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:57:20.809816 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.809768    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.309746344 +0000 UTC m=+1.742510218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:20.810090 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809876    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.810090 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809949    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.810090 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.809996    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.810239 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810202    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.810289 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810256    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.810365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.810365 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810333    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-grpc-tls\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.810448 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810393    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.810448 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810434    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.810536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810467    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.810536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810527    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.810612 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810558    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.810612 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810588    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.810737 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810615    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.810737 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810697    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.810737 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810732    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:20.810849 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810763    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.810849 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810794    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.810849 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810825    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.810972 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810852    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.810972 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810881    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.810972 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.810912    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.811101 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811028    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.811101 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4xp\" (UniqueName: \"kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:20.811101 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811094    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgc4\" (UniqueName: \"kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.811221 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811125    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.811221 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811160    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ctg\" (UniqueName: \"kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.811221 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811192    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.811366 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811223    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.811366 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811247    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.811366 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811281    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.811366 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811341    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.811536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:20.811536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.811536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811434    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.811536 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811465    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9fj\" (UniqueName: \"kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:57:20.811708 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811625    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.811708 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811659    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nj72\" (UniqueName: \"kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.811708 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811691    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.811837 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811716    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.811837 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811747    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.811837 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811779    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.811837 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811811    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811842    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811868    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811901    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctkj\" (UniqueName: \"kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811935    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cxl\" (UniqueName: \"kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.811968    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.812005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812000    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812028    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812060    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812089    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812123    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812150    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812186    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.812246 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812220    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812252    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812284    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812326    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812358    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812466    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812501    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.812554 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812554    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.812881 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.812775    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:20.812881 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.812857    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:20.812881 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812862    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813038 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.812941    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f2a8f9c-d148-4051-855b-79b8da1b57dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.813112 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813075    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813172 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813120    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813221 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813170    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.813273 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813217    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-metrics-tls\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.813353 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813335    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813409 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813370    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.813463 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813401    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.813560 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813540    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.813645 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813622    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d3eea62-15e2-488a-a427-abb5e81d034b-metrics-certs\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:20.813910 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813119    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovnkube-script-lib\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813938    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813947    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-metrics-client-ca\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.812863    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.312845534 +0000 UTC m=+1.745609407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813949    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.813983 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.813951    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.814241 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.813993    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.313973696 +0000 UTC m=+1.746737573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:20.814241 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814064    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.814241 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814127    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.814391 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814250    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02f2149-de58-41d0-892a-77532f24a26b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.814391 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814252    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f128ed7-b717-4188-84f8-0ea3f320dee4-iptables-alerter-script\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.814644 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814619    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.814709 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814690    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-daemon-config\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.814760 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814704    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.814801 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814754    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.814801 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814779    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.814801 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814795    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.814895 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814842    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.814953 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814935    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.815005 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.814976    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.815037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815022    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.815037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815026    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-tmp-dir\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.815103 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815045    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78hq\" (UniqueName: \"kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.815159 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815143    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.815194 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815174    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.815223 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815194    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.815255 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815244    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.815350 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815326    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.815384 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815354    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-textfile\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.815384 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815373    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.815437 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815397    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.815437 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815424    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.815509 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815468    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2098dac-6268-493b-b74d-d93639ac7e8d-serviceca\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.815509 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815493    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.815599 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815470    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.815599 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815546    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:20.815692 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815665    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffb43b53-7a8a-44d5-a802-2eb2691e2758-ovn-node-metrics-cert\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.815741 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815708    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffb43b53-7a8a-44d5-a802-2eb2691e2758-env-overrides\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.815849 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.815832    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/17f2a963-5150-4193-826b-bfd5a336ce5b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:20.827932 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.827895    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2nn\" (UniqueName: \"kubernetes.io/projected/992e92c9-6236-4e46-90bb-08f2a0b89a17-kube-api-access-bk2nn\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.828024 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.827968    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd62\" (UniqueName: \"kubernetes.io/projected/19df0167-7313-4ba8-a63f-99ce3cd97697-kube-api-access-9kd62\") pod \"network-check-target-qdhdb\" (UID: \"19df0167-7313-4ba8-a63f-99ce3cd97697\") " pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:20.828082 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.828048    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bfr\" (UniqueName: \"kubernetes.io/projected/17f2a963-5150-4193-826b-bfd5a336ce5b-kube-api-access-c9bfr\") pod \"monitoring-plugin-77f4f76594-n75wj\" (UID: \"17f2a963-5150-4193-826b-bfd5a336ce5b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:20.828113 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.828092    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrnm\" (UniqueName: \"kubernetes.io/projected/df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91-kube-api-access-8hrnm\") pod \"thanos-querier-86b57b86db-sjlhb\" (UID: \"df7eb1ed-eaa2-40cb-b4a6-92d6becc8d91\") " pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:20.828240 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.828219    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"alertmanager-main-0\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:20.828471 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.828457    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhtn\" (UniqueName: \"kubernetes.io/projected/c2098dac-6268-493b-b74d-d93639ac7e8d-kube-api-access-dxhtn\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.829164 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.829143    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqjh\" (UniqueName: \"kubernetes.io/projected/f6830ae7-bf50-490b-bbf8-f5d32835edfb-kube-api-access-bmqjh\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.829284 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.829267    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87gk\" (UniqueName: \"kubernetes.io/projected/6528c489-7d36-4881-a608-11ecdd2bf3f3-kube-api-access-j87gk\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.829881 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.829864    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9nd\" (UniqueName: \"kubernetes.io/projected/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kube-api-access-xf9nd\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.831520 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.831501    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wv7\" (UniqueName: \"kubernetes.io/projected/2f128ed7-b717-4188-84f8-0ea3f320dee4-kube-api-access-c7wv7\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.831765 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.831748    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8ls\" (UniqueName: \"kubernetes.io/projected/358670d6-0bd0-4c3b-81cb-beafe2872176-kube-api-access-ch8ls\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.834167 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.834149    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:20.834223 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.834174    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:20.834223 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.834188    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:20.834279 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:20.834259    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:21.33423961 +0000 UTC m=+1.767003503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:20.834428 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.834414    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9fj\" (UniqueName: \"kubernetes.io/projected/10f67d3c-20ef-48ee-b6a5-c33e841afd3d-kube-api-access-tj9fj\") pod \"ingress-canary-7r8w2\" (UID: \"10f67d3c-20ef-48ee-b6a5-c33e841afd3d\") " pod="openshift-ingress-canary/ingress-canary-7r8w2"
Oct 30 06:57:20.836321 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.836283    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ctg\" (UniqueName: \"kubernetes.io/projected/01063b0f-0dd4-40ac-abe3-edda6dcda17c-kube-api-access-m4ctg\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.837854 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.837831    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nj72\" (UniqueName: \"kubernetes.io/projected/42f96032-aac6-45a7-b77b-e7f43f33afbc-kube-api-access-5nj72\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.837935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.837857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgc4\" (UniqueName: \"kubernetes.io/projected/7de252f7-3dda-4a15-abb2-a69f2b22fb7e-kube-api-access-2fgc4\") pod \"dns-default-wmj4w\" (UID: \"7de252f7-3dda-4a15-abb2-a69f2b22fb7e\") " pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:20.837935 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.837878    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4xp\" (UniqueName: \"kubernetes.io/projected/1d3eea62-15e2-488a-a427-abb5e81d034b-kube-api-access-2g4xp\") pod \"network-metrics-daemon-vtjnb\" (UID: \"1d3eea62-15e2-488a-a427-abb5e81d034b\") " pod="openshift-multus/network-metrics-daemon-vtjnb"
Oct 30 06:57:20.838018 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.837970    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cxl\" (UniqueName: \"kubernetes.io/projected/ffb43b53-7a8a-44d5-a802-2eb2691e2758-kube-api-access-n5cxl\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.838062 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.838032    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctkj\" (UniqueName: \"kubernetes.io/projected/0f2a8f9c-d148-4051-855b-79b8da1b57dc-kube-api-access-bctkj\") pod \"kube-state-metrics-5dc8d7d89b-8khnk\" (UID: \"0f2a8f9c-d148-4051-855b-79b8da1b57dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8khnk"
Oct 30 06:57:20.838097 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.838069    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78hq\" (UniqueName: \"kubernetes.io/projected/f02f2149-de58-41d0-892a-77532f24a26b-kube-api-access-v78hq\") pod \"openshift-state-metrics-7bd7489957-vjt6z\" (UID: \"f02f2149-de58-41d0-892a-77532f24a26b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-vjt6z"
Oct 30 06:57:20.916209 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916122    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916209 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916164    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.916209 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916181    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916220    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916230    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916246    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2098dac-6268-493b-b74d-d93639ac7e8d-host\") pod \"node-ca-nxvzr\" (UID: \"c2098dac-6268-493b-b74d-d93639ac7e8d\") " pod="openshift-image-registry/node-ca-nxvzr"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916255    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916278    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-bin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916311    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916332    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-socket-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916343    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916365    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-kubelet-config\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916379    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916369    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916425    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916396    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-netd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916472 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916452    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916479    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916456    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-modprobe-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916490    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f128ed7-b717-4188-84f8-0ea3f320dee4-host-slash\") pod \"iptables-alerter-hr547\" (UID: \"2f128ed7-b717-4188-84f8-0ea3f320dee4\") " pod="openshift-network-operator/iptables-alerter-hr547"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-sys\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916511    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-kubernetes\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916535    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916562    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916588    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916607    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-multus-certs\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916619    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-registration-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916626    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-hostroot\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916627    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916652    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-var-lib-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916669    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-root\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916670    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916701    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-conf-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916727    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916793    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916809    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-etc-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916820    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916852    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-device-dir\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916862    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916886    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-d\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916894    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-etc-selinux\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.916912 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916885    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916929    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-cni-multus\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916940    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-system-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916972    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.916992    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917007    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917029    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917051    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-openvswitch\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917053    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917087    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917060    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-ovn\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-systemd\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917091    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-cnibin\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917134    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-slash\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917137    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-node-exporter-wtmp\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917164    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-run-netns\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917217    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917242    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917270    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917289    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysconfig\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917311    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917310    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-netns\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917318    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917335    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917339    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6528c489-7d36-4881-a608-11ecdd2bf3f3-hosts-file\") pod \"node-resolver-qgqkv\" (UID: \"6528c489-7d36-4881-a608-11ecdd2bf3f3\") " pod="openshift-dns/node-resolver-qgqkv"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917370    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917422    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917434    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917489    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917494    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-run-k8s-cni-cncf-io\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917502    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-etc-sysctl-conf\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917500    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2ab3bbf-b449-4b3c-b048-5180eb3b2e37-dbus\") pod \"global-pull-secret-syncer-f2pl2\" (UID: \"e2ab3bbf-b449-4b3c-b048-5180eb3b2e37\") " pod="kube-system/global-pull-secret-syncer-f2pl2"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917514    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917547    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5eb0f2a4-b0bb-4259-9f93-0e2331129305-sys-fs\") pod \"aws-ebs-csi-driver-node-828pv\" (UID: \"5eb0f2a4-b0bb-4259-9f93-0e2331129305\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828pv"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917556    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-host\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917583    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917611    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917645    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-host-var-lib-kubelet\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917658    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917687    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917660    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-var-lib-kubelet\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917689    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-systemd-units\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917719    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917733    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6830ae7-bf50-490b-bbf8-f5d32835edfb-sys\") pod \"node-exporter-vns7r\" (UID: \"f6830ae7-bf50-490b-bbf8-f5d32835edfb\") " pod="openshift-monitoring/node-exporter-vns7r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917695    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-cnibin\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917747    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917764    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-socket-dir-parent\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917786    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-etc-kubernetes\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917830    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/358670d6-0bd0-4c3b-81cb-beafe2872176-crio-socket\") pod \"insights-runtime-extractor-rqlss\" (UID: \"358670d6-0bd0-4c3b-81cb-beafe2872176\") " pod="openshift-insights/insights-runtime-extractor-rqlss"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917832    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917858    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01063b0f-0dd4-40ac-abe3-edda6dcda17c-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2gn\" (UID: \"01063b0f-0dd4-40ac-abe3-edda6dcda17c\") " pod="openshift-multus/multus-additional-cni-plugins-wp2gn"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917862    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917921    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/992e92c9-6236-4e46-90bb-08f2a0b89a17-multus-cni-dir\") pod \"multus-9mh4r\" (UID: \"992e92c9-6236-4e46-90bb-08f2a0b89a17\") " pod="openshift-multus/multus-9mh4r"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917937    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.917947 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917972    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.917989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918028    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-kubelet\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918029    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-run-systemd\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918042    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-run\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918047    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918090    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918105    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918111    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-host-cni-bin\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918138    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-node-log\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918152    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffb43b53-7a8a-44d5-a802-2eb2691e2758-log-socket\") pod \"ovnkube-node-6jgfh\" (UID: \"ffb43b53-7a8a-44d5-a802-2eb2691e2758\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:20.919614 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:20.918253    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f96032-aac6-45a7-b77b-e7f43f33afbc-lib-modules\") pod \"tuned-gc6hx\" (UID: \"42f96032-aac6-45a7-b77b-e7f43f33afbc\") " pod="openshift-cluster-node-tuning-operator/tuned-gc6hx"
Oct 30 06:57:21.203681 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.203602    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wmj4w"
Oct 30 06:57:21.321609 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321571    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321609 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321613    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321644    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321692    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321697    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321739    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.321754    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321785    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321816    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321801    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.321780606 +0000 UTC m=+2.754544498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321837    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:21.321851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321856    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:21.322232 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321871    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.32184705 +0000 UTC m=+2.754610924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:21.322232 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321889    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.321881511 +0000 UTC m=+2.754645381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:21.322232 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321907    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.321902803 +0000 UTC m=+2.754666706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:21.322232 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321921    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.321913954 +0000 UTC m=+2.754677825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:21.322232 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.321997    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.321980486 +0000 UTC m=+2.754744371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:21.422230 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:21.422187    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:21.422405 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.422367    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:21.422405 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.422388    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:21.422405 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.422399    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:21.422521 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:21.422446    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:22.422434096 +0000 UTC m=+2.855197991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:22.066693 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.066654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:22.173443 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.173414    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:22.330952 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.330870    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.330952 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.330933    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.330982    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.331010    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331027    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.331046    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.331088    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331099    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331130    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.33110842 +0000 UTC m=+4.763872304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331135    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:22.331148 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331154    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.331142002 +0000 UTC m=+4.763905887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331159    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331169    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331189    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.331175834 +0000 UTC m=+4.763939705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331140    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331214    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.331201252 +0000 UTC m=+4.763965123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331234    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.3312238 +0000 UTC m=+4.763987671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:22.331599 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.331245    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.331240276 +0000 UTC m=+4.764004147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:22.431922 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.431882    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:22.432084 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.432047    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:22.432084 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.432070    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:22.432084 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.432079    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:22.432218 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:22.432130    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:24.432115277 +0000 UTC m=+4.864879149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:22.572865 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.572830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:22.692378 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:22.692320    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:23.619866 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:23.619836    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86b57b86db-sjlhb"
Oct 30 06:57:24.350407 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350367    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350407 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350415    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350443    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350483    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350527    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350560    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350584    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.350570825 +0000 UTC m=+8.783334709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350603    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350604    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350631    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.350612942 +0000 UTC m=+8.783376831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:24.350642 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350565    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350662    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350684    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.350673132 +0000 UTC m=+8.783437017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350708    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.350697545 +0000 UTC m=+8.783461422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350722    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.35071654 +0000 UTC m=+8.783480411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.350776    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350853    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:24.350915 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.350885    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.350875598 +0000 UTC m=+8.783639472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:24.452031 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:24.451995    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:24.452123 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.452111    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:24.452170 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.452127    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:24.452170 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.452137    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:24.452224 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:24.452183    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:28.452171611 +0000 UTC m=+8.884935496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:25.772559 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:25.772531    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:26.251900 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:26.251867    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djbct"
Oct 30 06:57:26.783781 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:26.783745    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:26.926205 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:26.926163    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:28.391207 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391152    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391207 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391211    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391242    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391266    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391284    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391328    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.391344    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391352    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391403    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391406    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391416    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391453    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391411    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391390025 +0000 UTC m=+16.824153914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391545    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391524376 +0000 UTC m=+16.824288255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391565    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391552994 +0000 UTC m=+16.824316872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391579    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391571331 +0000 UTC m=+16.824335220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391595    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391586596 +0000 UTC m=+16.824350486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:28.391753 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.391609    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.391600251 +0000 UTC m=+16.824364126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:28.492603 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.492561    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:28.492790 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.492768    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:28.492851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.492800    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:28.492851 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.492814    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:28.492967 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:28.492885    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:36.492866474 +0000 UTC m=+16.925630368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:28.601952 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:28.601901    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:29.472386 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:29.472309    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n75wj"
Oct 30 06:57:30.817541 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:30.817514    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:30.818025 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:30.817842    6738 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:57:31.197678 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:31.197628    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:31.311341 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:31.311289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:36.097186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.097143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:36.213806 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.213766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:36.465722 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.465666    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.465722 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.465724    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.465967 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.465795    6738 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:36.465967 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.465867    6738 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:36.465967 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.465878    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.465863537 +0000 UTC m=+32.898627408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 06:57:36.465967 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.465926    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.465912982 +0000 UTC m=+32.898676857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 06:57:36.465967 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.465958    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.466014    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.466037    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.466067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466090    6738 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466112    6738 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466121    6738 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466129    6738 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 06:57:36.466141 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466142    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.466131813 +0000 UTC m=+32.898895688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 06:57:36.466406 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466161    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.46615126 +0000 UTC m=+32.898915137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-config" not registered
Oct 30 06:57:36.466406 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466177    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.466168154 +0000 UTC m=+32.898932031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 06:57:36.466406 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.466192    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.466183203 +0000 UTC m=+32.898947076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : object "openshift-console"/"service-ca" not registered
Oct 30 06:57:36.567135 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:36.567087    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"console-666d7bbf55-jm86p\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") " pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:36.567264 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.567246    6738 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 06:57:36.567320 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.567270    6738 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 06:57:36.567320 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.567280    6738 projected.go:196] Error preparing data for projected volume kube-api-access-dwjh4 for pod openshift-console/console-666d7bbf55-jm86p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:36.567386 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:36.567347    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4 podName:2f664d3f-d782-4676-8de4-0dbd66f2d5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:52.567334641 +0000 UTC m=+33.000098533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwjh4" (UniqueName: "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4") pod "console-666d7bbf55-jm86p" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 06:57:37.516580 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:37.516545    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 06:57:37.557229 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:37.557195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:37.669562 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:37.669523    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:38.974382 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:38.974349    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79c766bbf-t4dbg"]
Oct 30 06:57:39.017230 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.017194    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c766bbf-t4dbg"]
Oct 30 06:57:39.017409 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.017290    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.033062 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.033038    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 06:57:39.033062 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.033043    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 06:57:39.045778 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045697    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 06:57:39.045778 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045709    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 06:57:39.045778 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045699    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 06:57:39.045778 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045728    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vjmtx\""
Oct 30 06:57:39.045778 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045779    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 06:57:39.046035 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.045791    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 06:57:39.072335 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.072278    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 06:57:39.089649 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089619    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089805 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089653    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-oauth-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089805 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089683    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089805 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089704    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-service-ca\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089913 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089812    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a56b0f32-f8b9-4a41-90aa-7d900ef98531-tmp\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089913 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089852    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-oauth-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089913 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089875    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wml\" (UniqueName: \"kubernetes.io/projected/a56b0f32-f8b9-4a41-90aa-7d900ef98531-kube-api-access-w5wml\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.089913 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.089901    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-trusted-ca-bundle\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.123143 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.123111    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:39.190363 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190331    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190486 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190367    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-oauth-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190486 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190486 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190416    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-service-ca\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190486 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190470    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a56b0f32-f8b9-4a41-90aa-7d900ef98531-tmp\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190679 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190519    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-oauth-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190679 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190556    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wml\" (UniqueName: \"kubernetes.io/projected/a56b0f32-f8b9-4a41-90aa-7d900ef98531-kube-api-access-w5wml\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190679 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190592    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-trusted-ca-bundle\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.190855 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190831    6738 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 06:57:39.190951 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.190930    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a56b0f32-f8b9-4a41-90aa-7d900ef98531-tmp\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.191212 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.191193    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-oauth-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.191316 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.191210    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.191390 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.191285    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-service-ca\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.191609 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.191591    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a56b0f32-f8b9-4a41-90aa-7d900ef98531-trusted-ca-bundle\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.193174 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.193153    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-serving-cert\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.193264 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.193202    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a56b0f32-f8b9-4a41-90aa-7d900ef98531-console-oauth-config\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.214708 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.214669    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wml\" (UniqueName: \"kubernetes.io/projected/a56b0f32-f8b9-4a41-90aa-7d900ef98531-kube-api-access-w5wml\") pod \"console-79c766bbf-t4dbg\" (UID: \"a56b0f32-f8b9-4a41-90aa-7d900ef98531\") " pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.230798 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.230720    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:39.339685 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.339645    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:39.412026 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.411972    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:39.521215 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.521113    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:39.635889 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.635767    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c766bbf-t4dbg"]
Oct 30 06:57:39.638203 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:39.638165    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56b0f32_f8b9_4a41_90aa_7d900ef98531.slice/crio-8f247098e7437e50e7737973a56f21f459db3eff1f591be13f26775d0877285f WatchSource:0}: Error finding container 8f247098e7437e50e7737973a56f21f459db3eff1f591be13f26775d0877285f: Status 404 returned error can't find the container with id 8f247098e7437e50e7737973a56f21f459db3eff1f591be13f26775d0877285f
Oct 30 06:57:39.817711 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.817615    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c766bbf-t4dbg" event={"ID":"a56b0f32-f8b9-4a41-90aa-7d900ef98531","Type":"ContainerStarted","Data":"2aa43b92692a25f9a36c11efe45400ad345caf455613fdde47d183b704d21941"}
Oct 30 06:57:39.817711 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.817666    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c766bbf-t4dbg" event={"ID":"a56b0f32-f8b9-4a41-90aa-7d900ef98531","Type":"ContainerStarted","Data":"8f247098e7437e50e7737973a56f21f459db3eff1f591be13f26775d0877285f"}
Oct 30 06:57:39.861767 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:39.861706    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79c766bbf-t4dbg" podStartSLOduration=1.861689487 podStartE2EDuration="1.861689487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:39.861014848 +0000 UTC m=+20.293778741" watchObservedRunningTime="2026-10-30 06:57:39.861689487 +0000 UTC m=+20.294453383"
Oct 30 06:57:41.200571 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.200539    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:41.201001 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.200955    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="alertmanager" containerID="cri-o://9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc" gracePeriod=120
Oct 30 06:57:41.201069 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.200998    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-metric" containerID="cri-o://b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279" gracePeriod=120
Oct 30 06:57:41.201069 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.201050    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy" containerID="cri-o://7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2" gracePeriod=120
Oct 30 06:57:41.201175 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.201018    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="prom-label-proxy" containerID="cri-o://bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628" gracePeriod=120
Oct 30 06:57:41.201175 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.201046    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-web" containerID="cri-o://744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949" gracePeriod=120
Oct 30 06:57:41.201258 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.201206    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="config-reloader" containerID="cri-o://d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef" gracePeriod=120
Oct 30 06:57:41.662666 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.662630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:41.770277 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.770235    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:41.827927 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827889    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628" exitCode=0
Oct 30 06:57:41.827927 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827914    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279" exitCode=0
Oct 30 06:57:41.827927 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827921    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2" exitCode=0
Oct 30 06:57:41.827927 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827929    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef" exitCode=0
Oct 30 06:57:41.827927 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827934    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc" exitCode=0
Oct 30 06:57:41.828199 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.827960    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"}
Oct 30 06:57:41.828199 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.828025    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"}
Oct 30 06:57:41.828199 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.828050    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"}
Oct 30 06:57:41.828199 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.828074    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"}
Oct 30 06:57:41.828199 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.828096    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"}
Oct 30 06:57:41.847831 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:41.847807    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jgfh"
Oct 30 06:57:42.439923 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.439897    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:42.523003 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.522970    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523198 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523038    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523198 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523136    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523344 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523212    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523400 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523338    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523400 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523383    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523495 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523414    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523495 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523461    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523591 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523502    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523591 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523510    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:42.523591 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523534    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523591 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523577    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523781 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523615    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.523781 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523643    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") pod \"fa866eab-cad8-49ed-9c13-52a4f33605a8\" (UID: \"fa866eab-cad8-49ed-9c13-52a4f33605a8\") "
Oct 30 06:57:42.524214 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.523958    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:42.524214 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.524111    6738 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-main-db\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.524214 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.524134    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-metrics-client-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.524589 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.524515    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:42.526995 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.526851    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.526995 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.526859    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:42.526995 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.526906    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "kube-api-access-b276w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:42.527386 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.527355    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.527499 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.527459    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:42.527566 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.527493    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.527622 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.527588    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.527732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.527717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.536932 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.536873    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.543147 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.543119    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config" pod "fa866eab-cad8-49ed-9c13-52a4f33605a8" (UID: "fa866eab-cad8-49ed-9c13-52a4f33605a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:42.624833 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624790    6738 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.624833 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624823    6738 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa866eab-cad8-49ed-9c13-52a4f33605a8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.624833 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624834    6738 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-web-config\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.624833 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624843    6738 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624853    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b276w\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-kube-api-access-b276w\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624861    6738 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa866eab-cad8-49ed-9c13-52a4f33605a8-tls-assets\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624870    6738 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-main-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624879    6738 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624888    6738 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa866eab-cad8-49ed-9c13-52a4f33605a8-config-out\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624896    6738 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.625127 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.624905    6738 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa866eab-cad8-49ed-9c13-52a4f33605a8-cluster-tls-config\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:42.835213 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.835120    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerID="744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949" exitCode=0
Oct 30 06:57:42.835356 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.835207    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"}
Oct 30 06:57:42.835356 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.835223    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:42.835356 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.835281    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa866eab-cad8-49ed-9c13-52a4f33605a8","Type":"ContainerDied","Data":"6941cfde893bd7a3fd9eb2c999a1ae95dbbacd722641ab12b2e307f0e3b37763"}
Oct 30 06:57:42.835356 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.835319    6738 scope.go:122] "RemoveContainer" containerID="bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"
Oct 30 06:57:42.843505 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.843485    6738 scope.go:122] "RemoveContainer" containerID="b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"
Oct 30 06:57:42.850387 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.850363    6738 scope.go:122] "RemoveContainer" containerID="7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"
Oct 30 06:57:42.857714 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.857695    6738 scope.go:122] "RemoveContainer" containerID="744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"
Oct 30 06:57:42.864063 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.864042    6738 scope.go:122] "RemoveContainer" containerID="d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"
Oct 30 06:57:42.871091 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.871073    6738 scope.go:122] "RemoveContainer" containerID="9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"
Oct 30 06:57:42.878205 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.878183    6738 scope.go:122] "RemoveContainer" containerID="fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a"
Oct 30 06:57:42.885148 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885132    6738 scope.go:122] "RemoveContainer" containerID="bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"
Oct 30 06:57:42.885443 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.885424    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628\": container with ID starting with bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628 not found: ID does not exist" containerID="bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"
Oct 30 06:57:42.885498 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885453    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628"} err="failed to get container status \"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628\": rpc error: code = NotFound desc = could not find container \"bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628\": container with ID starting with bb8f1efab35f0843d2795eb210f3f48ae5451534fb7a28aac0f88a89a5177628 not found: ID does not exist"
Oct 30 06:57:42.885498 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885488    6738 scope.go:122] "RemoveContainer" containerID="b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"
Oct 30 06:57:42.885731 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.885710    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279\": container with ID starting with b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279 not found: ID does not exist" containerID="b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"
Oct 30 06:57:42.885823 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885736    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279"} err="failed to get container status \"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279\": rpc error: code = NotFound desc = could not find container \"b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279\": container with ID starting with b6d23e745711ae99d5a10d7ece8bf90e0a292bfa980983c966bd107edff80279 not found: ID does not exist"
Oct 30 06:57:42.885823 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885752    6738 scope.go:122] "RemoveContainer" containerID="7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"
Oct 30 06:57:42.885978 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.885964    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2\": container with ID starting with 7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2 not found: ID does not exist" containerID="7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"
Oct 30 06:57:42.886025 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885980    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2"} err="failed to get container status \"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2\": rpc error: code = NotFound desc = could not find container \"7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2\": container with ID starting with 7e88822615c096414f0dcfd653992b5e52a71f3f20a78f3c602d83be23a31ea2 not found: ID does not exist"
Oct 30 06:57:42.886025 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.885991    6738 scope.go:122] "RemoveContainer" containerID="744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"
Oct 30 06:57:42.886210 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.886194    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949\": container with ID starting with 744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949 not found: ID does not exist" containerID="744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"
Oct 30 06:57:42.886248 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.886216    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949"} err="failed to get container status \"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949\": rpc error: code = NotFound desc = could not find container \"744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949\": container with ID starting with 744f788b445def6ba7451f3d919d7ccc89c64624e8a127d10fe7aacc4b6e6949 not found: ID does not exist"
Oct 30 06:57:42.886248 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.886230    6738 scope.go:122] "RemoveContainer" containerID="d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"
Oct 30 06:57:42.886537 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.886520    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef\": container with ID starting with d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef not found: ID does not exist" containerID="d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"
Oct 30 06:57:42.886604 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.886546    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef"} err="failed to get container status \"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef\": rpc error: code = NotFound desc = could not find container \"d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef\": container with ID starting with d3f529e8bc73ea13054fac3187eba42ddf7bbb91f0b3309744725567f7d221ef not found: ID does not exist"
Oct 30 06:57:42.886604 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.886565    6738 scope.go:122] "RemoveContainer" containerID="9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"
Oct 30 06:57:42.886828 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.886813    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc\": container with ID starting with 9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc not found: ID does not exist" containerID="9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"
Oct 30 06:57:42.890700 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.887180    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc"} err="failed to get container status \"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc\": rpc error: code = NotFound desc = could not find container \"9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc\": container with ID starting with 9d7b9c828c87e46d5b73476bb9891518b2e5f2e1e5627240b622d7f603f2cafc not found: ID does not exist"
Oct 30 06:57:42.890700 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.887223    6738 scope.go:122] "RemoveContainer" containerID="fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a"
Oct 30 06:57:42.890964 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:57:42.890935    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a\": container with ID starting with fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a not found: ID does not exist" containerID="fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a"
Oct 30 06:57:42.891032 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.891000    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a"} err="failed to get container status \"fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a\": rpc error: code = NotFound desc = could not find container \"fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a\": container with ID starting with fe61883c7d92f18f8dc987a9ea4ce801a8be8a8e70756710caf465d0678fcb5a not found: ID does not exist"
Oct 30 06:57:42.892075 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.892058    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:42.904515 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.904490    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:42.926590 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926563    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:42.926884 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926869    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="init-config-reloader"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926886    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="init-config-reloader"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926906    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="alertmanager"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926913    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="alertmanager"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926930    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-metric"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926938    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-metric"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926951    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="prom-label-proxy"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926956    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="prom-label-proxy"
Oct 30 06:57:42.926960 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926962    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-web"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926966    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-web"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926980    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="config-reloader"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926988    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="config-reloader"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.926997    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927001    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927046    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="alertmanager"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927057    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-metric"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927069    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="prom-label-proxy"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927076    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy-web"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927082    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="config-reloader"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927088    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="kube-rbac-proxy"
Oct 30 06:57:42.927236 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.927094    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" containerName="init-config-reloader"
Oct 30 06:57:42.931076 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.931056    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:42.946910 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946883    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 06:57:42.946910 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946889    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 06:57:42.946910 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946894    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 06:57:42.947106 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946924    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 06:57:42.947106 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946944    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 06:57:42.947106 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.946979    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 06:57:42.947106 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.947010    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 06:57:42.947106 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.947027    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 06:57:42.947261 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.947151    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b8vbj\""
Oct 30 06:57:42.960050 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.959758    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 06:57:42.960601 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:42.960578    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:43.027033 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.026988    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-out\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027033 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027034    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tn2t\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-kube-api-access-7tn2t\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027328 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027162    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027328 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027214    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027328 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027276    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-volume\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027328 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027321    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027545 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027362    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-web-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027545 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027466    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027545 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027503    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027690 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027553    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027690 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027581    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027690 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027614    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.027690 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.027674    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128528 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128488    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128531    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128582    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128604    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128633    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128662    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.128732 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128717    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-out\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128747    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tn2t\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-kube-api-access-7tn2t\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128801    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128834    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128892    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-volume\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128918    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.129037 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.128958    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-web-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.130548 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.129536    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.130548 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.129620    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.130548 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.129783    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.131693 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.131665    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.131789 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.131773    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.131862 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.131841    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-out\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.132006 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.131982    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.132150 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.132132    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-web-config\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.132218 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.132199    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.132379 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.132363    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.132422 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.132401    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-config-volume\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.133173 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.133153    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.153562 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.153539    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tn2t\" (UniqueName: \"kubernetes.io/projected/aab7e92d-aa0e-4568-9c8f-7d76b5968cfd-kube-api-access-7tn2t\") pod \"alertmanager-main-0\" (UID: \"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.248844 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.248805    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 06:57:43.579251 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.579167    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 06:57:43.592817 ip-10-0-139-124 kubenswrapper[6738]: W1030 06:57:43.592788    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab7e92d_aa0e_4568_9c8f_7d76b5968cfd.slice/crio-71418a6b5476561aba0bc69282de84c4004ac1fd4348b444c01bc80d3b71ad35 WatchSource:0}: Error finding container 71418a6b5476561aba0bc69282de84c4004ac1fd4348b444c01bc80d3b71ad35: Status 404 returned error can't find the container with id 71418a6b5476561aba0bc69282de84c4004ac1fd4348b444c01bc80d3b71ad35
Oct 30 06:57:43.686931 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.686902    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa866eab-cad8-49ed-9c13-52a4f33605a8" path="/var/lib/kubelet/pods/fa866eab-cad8-49ed-9c13-52a4f33605a8/volumes"
Oct 30 06:57:43.840564 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.840475    6738 generic.go:356] "Generic (PLEG): container finished" podID="aab7e92d-aa0e-4568-9c8f-7d76b5968cfd" containerID="a59af957fca1d22b3f276934e9d9d8319965c7e532275a2f34e2841830aced31" exitCode=0
Oct 30 06:57:43.840720 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.840556    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerDied","Data":"a59af957fca1d22b3f276934e9d9d8319965c7e532275a2f34e2841830aced31"}
Oct 30 06:57:43.840720 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:43.840626    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"71418a6b5476561aba0bc69282de84c4004ac1fd4348b444c01bc80d3b71ad35"}
Oct 30 06:57:44.848192 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848159    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"efdfcdbfd713c09cd82843f0a58f38b2432a67b680fcc8041f680c510f82e2d8"}
Oct 30 06:57:44.848581 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"1dd0e9bbb6aa573ef040082fabe915f906eeefa7fba93cc05d9688960c734c61"}
Oct 30 06:57:44.848581 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848288    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"fb99b372039fcfc7c4db024ac52f81f683e231084bdcd556f75794d214dbda1a"}
Oct 30 06:57:44.848581 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"38f8afb65b7152b1818ff96fd662fe9433c29c4c4439363a5688168d37c5b68d"}
Oct 30 06:57:44.848581 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848365    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"c8076171389d3a02c6c2089efd6963418195ea1b6904ff679c3a6ed822bf6fc9"}
Oct 30 06:57:44.848581 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.848388    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aab7e92d-aa0e-4568-9c8f-7d76b5968cfd","Type":"ContainerStarted","Data":"8177cb79b4543fd1eb1db6b1c36307efd13722c65401a64fe113ba122774dd10"}
Oct 30 06:57:44.915115 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:44.915055    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.903428606 podStartE2EDuration="2.9150427s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.614094ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:44.914070545 +0000 UTC m=+25.346834439" watchObservedRunningTime="2026-10-30 06:57:44.9150427 +0000 UTC m=+25.347806584"
Oct 30 06:57:45.721848 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:45.721774    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-666d7bbf55-jm86p" podUID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" containerName="console" containerID="cri-o://29697f8de841c49547738e68dd8be773a9c031e3c39a2fbf74876cf8bba60f5b" gracePeriod=15
Oct 30 06:57:45.854959 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:45.854930    6738 generic.go:356] "Generic (PLEG): container finished" podID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" containerID="29697f8de841c49547738e68dd8be773a9c031e3c39a2fbf74876cf8bba60f5b" exitCode=0
Oct 30 06:57:45.855341 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:45.855001    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666d7bbf55-jm86p" event={"ID":"2f664d3f-d782-4676-8de4-0dbd66f2d5c6","Type":"ContainerDied","Data":"29697f8de841c49547738e68dd8be773a9c031e3c39a2fbf74876cf8bba60f5b"}
Oct 30 06:57:45.967267 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:45.967243    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:46.060712 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060618    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060712 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060696    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060728    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060757    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060815    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060856    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060902    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.060954 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060931    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:46.061238 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.060948    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") pod \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\" (UID: \"2f664d3f-d782-4676-8de4-0dbd66f2d5c6\") "
Oct 30 06:57:46.061238 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061169    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.061386 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061267    6738 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.061386 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061270    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.061386 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061283    6738 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-tmp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.061525 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.061577 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.061521    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.063186 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.063156    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.063327 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.063199    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.063327 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.063205    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4" pod "2f664d3f-d782-4676-8de4-0dbd66f2d5c6" (UID: "2f664d3f-d782-4676-8de4-0dbd66f2d5c6"). InnerVolumeSpecName "kube-api-access-dwjh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:46.162733 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162692    6738 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-oauth-config\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.162733 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162735    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjh4\" (UniqueName: \"kubernetes.io/projected/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-kube-api-access-dwjh4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.162917 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162752    6738 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-serving-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.162917 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162766    6738 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-trusted-ca-bundle\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.162917 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162779    6738 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-oauth-serving-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.162917 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.162793    6738 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f664d3f-d782-4676-8de4-0dbd66f2d5c6-console-config\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.860653 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.860627    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666d7bbf55-jm86p"
Oct 30 06:57:46.861170 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.860619    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666d7bbf55-jm86p" event={"ID":"2f664d3f-d782-4676-8de4-0dbd66f2d5c6","Type":"ContainerDied","Data":"1d4e2ed1e0d2b9a54d6c95c3a109dfde5dfb6795e1dd2f8ea37016c2870d07d1"}
Oct 30 06:57:46.861170 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.860762    6738 scope.go:122] "RemoveContainer" containerID="29697f8de841c49547738e68dd8be773a9c031e3c39a2fbf74876cf8bba60f5b"
Oct 30 06:57:46.913163 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.913129    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:46.925824 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:46.925792    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666d7bbf55-jm86p"]
Oct 30 06:57:47.689858 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:47.686758    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" path="/var/lib/kubelet/pods/2f664d3f-d782-4676-8de4-0dbd66f2d5c6/volumes"
Oct 30 06:57:48.348907 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:48.348866    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:48.484781 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:48.484735    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:48.604505 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:48.604428    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qdhdb"
Oct 30 06:57:49.340727 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:49.340686    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:49.340916 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:49.340750    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:49.345634 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:49.345610    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:49.874744 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:49.874717    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79c766bbf-t4dbg"
Oct 30 06:57:50.037910 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:50.037870    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:52.559442 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:52.559405    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:57:52.679500 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:57:52.679466    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:04.382169 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:04.382132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:04.494624 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:04.494581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:07.516633 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:58:07.516599    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 06:58:21.761397 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:21.761358    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:21.905946 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:21.905907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:26.906865 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:26.906830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:37.517186 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:58:37.517150    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 06:58:43.231764 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:43.231727    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:58:43.348139 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:58:43.348102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:59:07.517441 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:59:07.517362    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 06:59:27.035917 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:59:27.035882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:59:37.517186 ip-10-0-139-124 kubenswrapper[6738]: E1030 06:59:37.517145    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 06:59:59.565555 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:59:59.565517    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:59:59.735745 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:59:59.735711    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 06:59:59.860838 ip-10-0-139-124 kubenswrapper[6738]: I1030 06:59:59.860738    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:00.528143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.528106    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"]
Oct 30 07:00:00.528517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.528500    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" containerName="console"
Oct 30 07:00:00.528609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.528520    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" containerName="console"
Oct 30 07:00:00.528609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.528593    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f664d3f-d782-4676-8de4-0dbd66f2d5c6" containerName="console"
Oct 30 07:00:00.530624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.530601    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.552503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.552471    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-ktnjf\""
Oct 30 07:00:00.552664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.552549    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 07:00:00.565449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.565420    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 07:00:00.566108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.566091    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"]
Oct 30 07:00:00.583774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.583746    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.583944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.583803    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.583944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.583909    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmrc\" (UniqueName: \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.606420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.606385    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"]
Oct 30 07:00:00.608842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.608820    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.631133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.631103    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"]
Oct 30 07:00:00.656100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.656065    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-clvkk\""
Oct 30 07:00:00.684793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.684757    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.684957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.684799    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdwq\" (UniqueName: \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.684957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.684840    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.684957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.684860    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.684957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.684880    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.685104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.685059    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmrc\" (UniqueName: \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.685177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.685150    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.685252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.685242    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.715461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.715422    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmrc\" (UniqueName: \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.785505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.785404    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdwq\" (UniqueName: \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.785505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.785477    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.785760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.785506    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.785921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.785901    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.785961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.785928    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.809930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.809889    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdwq\" (UniqueName: \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.843751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.843719    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:00.921983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.921942    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:00.996172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.996141    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:00:00.999998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:00.998652    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.017924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.017892    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 07:00:01.018454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018412    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 07:00:01.018454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018426    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 07:00:01.018454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018433    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 07:00:01.018682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018412    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 07:00:01.018682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018465    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjg2r\""
Oct 30 07:00:01.018682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.018470    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 07:00:01.023652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.023620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:01.026833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.026811    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:00:01.087088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.086988    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087034    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087050    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087166    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl9t\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087215    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087273    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.087494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.087318    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.136370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.136334    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:01.139984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.139957    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"]
Oct 30 07:00:01.142583 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:00:01.142553    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529b7749_5afe_410a_bbd1_e77532149759.slice/crio-fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f WatchSource:0}: Error finding container fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f: Status 404 returned error can't find the container with id fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f
Oct 30 07:00:01.144587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.144572    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:00:01.188627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188595    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188632    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188661    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188689    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188705    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188730    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl9t\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.188863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.188778    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.189498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.189446    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.191200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.191166    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.191374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.191355    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.191436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.191421    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.191479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.191425    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.212927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.212894    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.213066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.212934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl9t\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\") pod \"istiod-openshift-gateway-7876bfcb74-lvh8q\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.232688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.232663    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"]
Oct 30 07:00:01.234562 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:00:01.234533    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72674139_4de1_4ee5_af29_d53a169b27c0.slice/crio-7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e WatchSource:0}: Error finding container 7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e: Status 404 returned error can't find the container with id 7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e
Oct 30 07:00:01.310557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.310518    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:01.319590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.319559    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f" event={"ID":"72674139-4de1-4ee5-af29-d53a169b27c0","Type":"ContainerStarted","Data":"7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e"}
Oct 30 07:00:01.320675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.320652    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7" event={"ID":"529b7749-5afe-410a-bbd1-e77532149759","Type":"ContainerStarted","Data":"fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f"}
Oct 30 07:00:01.620409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:01.620385    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:00:01.622184 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:00:01.622156    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e1ac56_e0e1_45fa_84c7_f5ac047027ff.slice/crio-335c1e25e623cd85c3fccd15781a75e6afd9cd9987dcd438988957d04ff47737 WatchSource:0}: Error finding container 335c1e25e623cd85c3fccd15781a75e6afd9cd9987dcd438988957d04ff47737: Status 404 returned error can't find the container with id 335c1e25e623cd85c3fccd15781a75e6afd9cd9987dcd438988957d04ff47737
Oct 30 07:00:02.297428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:02.297382    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:02.326777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:02.326721    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" event={"ID":"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff","Type":"ContainerStarted","Data":"335c1e25e623cd85c3fccd15781a75e6afd9cd9987dcd438988957d04ff47737"}
Oct 30 07:00:05.019800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.019756    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:00:05.020252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.019842    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:00:05.346667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.346009    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" event={"ID":"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff","Type":"ContainerStarted","Data":"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"}
Oct 30 07:00:05.346667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.346382    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:05.407670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.407608    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" podStartSLOduration=2.013147643 podStartE2EDuration="5.407586217s" totalImagesPullingTime="3.394438574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:05.406825888 +0000 UTC m=+165.839589785" watchObservedRunningTime="2026-10-30 07:00:05.407586217 +0000 UTC m=+165.840350110"
Oct 30 07:00:05.830957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.830922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:05.954389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:05.954353    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:06.352507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:06.352470    6738 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lvh8q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:00:06.352979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:06.352536    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:00:07.191005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:07.190955    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:07.354145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:07.354104    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:07.467644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:07.467557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:07.531615 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:00:07.531577    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:00:08.362066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:08.362031    6738 generic.go:356] "Generic (PLEG): container finished" podID="529b7749-5afe-410a-bbd1-e77532149759" containerID="32ea53d5f83f2de2f269e177ba1155d4dcf8a2e3b2db571ba53e3b1b88e4f6dc" exitCode=0
Oct 30 07:00:08.362569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:08.362111    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7" event={"ID":"529b7749-5afe-410a-bbd1-e77532149759","Type":"ContainerDied","Data":"32ea53d5f83f2de2f269e177ba1155d4dcf8a2e3b2db571ba53e3b1b88e4f6dc"}
Oct 30 07:00:09.352118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:09.352083    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:00:09.368552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:09.368511    6738 generic.go:356] "Generic (PLEG): container finished" podID="72674139-4de1-4ee5-af29-d53a169b27c0" containerID="51b786e38a2866c75c378d4ee8d5a578fbad52ea7d73afc6525f97a27004a1be" exitCode=0
Oct 30 07:00:09.368982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:09.368599    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f" event={"ID":"72674139-4de1-4ee5-af29-d53a169b27c0","Type":"ContainerDied","Data":"51b786e38a2866c75c378d4ee8d5a578fbad52ea7d73afc6525f97a27004a1be"}
Oct 30 07:00:10.543082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:10.542987    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:11.378982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:11.378949    6738 generic.go:356] "Generic (PLEG): container finished" podID="529b7749-5afe-410a-bbd1-e77532149759" containerID="92a836999bff6d6f660fc93c5cbea063a69db763cef67684ec72ad687f9c5786" exitCode=0
Oct 30 07:00:11.379175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:11.379006    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7" event={"ID":"529b7749-5afe-410a-bbd1-e77532149759","Type":"ContainerDied","Data":"92a836999bff6d6f660fc93c5cbea063a69db763cef67684ec72ad687f9c5786"}
Oct 30 07:00:12.386287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:12.386250    6738 generic.go:356] "Generic (PLEG): container finished" podID="72674139-4de1-4ee5-af29-d53a169b27c0" containerID="bc19d43ad4e327132eda025e9c9cb1c08f80d8938e67fd760d608e6ff6f02c0a" exitCode=0
Oct 30 07:00:12.386764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:12.386395    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f" event={"ID":"72674139-4de1-4ee5-af29-d53a169b27c0","Type":"ContainerDied","Data":"bc19d43ad4e327132eda025e9c9cb1c08f80d8938e67fd760d608e6ff6f02c0a"}
Oct 30 07:00:16.723700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:16.723663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:18.416697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:18.416646    6738 generic.go:356] "Generic (PLEG): container finished" podID="529b7749-5afe-410a-bbd1-e77532149759" containerID="9b369a697bcc1fe25a29e938b2d308dba8d7742fcdb8aa28b85dafc8f209638e" exitCode=0
Oct 30 07:00:18.417154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:18.416757    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7" event={"ID":"529b7749-5afe-410a-bbd1-e77532149759","Type":"ContainerDied","Data":"9b369a697bcc1fe25a29e938b2d308dba8d7742fcdb8aa28b85dafc8f209638e"}
Oct 30 07:00:19.423144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.423113    6738 generic.go:356] "Generic (PLEG): container finished" podID="72674139-4de1-4ee5-af29-d53a169b27c0" containerID="87740edf3861b255880b558046270d769b34deeab015fccbf251c5673dcc652d" exitCode=0
Oct 30 07:00:19.423559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.423202    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f" event={"ID":"72674139-4de1-4ee5-af29-d53a169b27c0","Type":"ContainerDied","Data":"87740edf3861b255880b558046270d769b34deeab015fccbf251c5673dcc652d"}
Oct 30 07:00:19.547201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.547170    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:19.656086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.656052    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\" (UniqueName: \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\") pod \"529b7749-5afe-410a-bbd1-e77532149759\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") "
Oct 30 07:00:19.656249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.656119    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\") pod \"529b7749-5afe-410a-bbd1-e77532149759\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") "
Oct 30 07:00:19.656249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.656210    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\") pod \"529b7749-5afe-410a-bbd1-e77532149759\" (UID: \"529b7749-5afe-410a-bbd1-e77532149759\") "
Oct 30 07:00:19.656568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.656534    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle" pod "529b7749-5afe-410a-bbd1-e77532149759" (UID: "529b7749-5afe-410a-bbd1-e77532149759"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:19.658261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.658236    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc" pod "529b7749-5afe-410a-bbd1-e77532149759" (UID: "529b7749-5afe-410a-bbd1-e77532149759"). InnerVolumeSpecName "kube-api-access-qlmrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:19.660321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.660284    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util" pod "529b7749-5afe-410a-bbd1-e77532149759" (UID: "529b7749-5afe-410a-bbd1-e77532149759"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:19.678802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.678718    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:19.757273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.757243    6738 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-util\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:19.757273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.757269    6738 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/529b7749-5afe-410a-bbd1-e77532149759-bundle\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:19.757273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.757278    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmrc\" (UniqueName: \"kubernetes.io/projected/529b7749-5afe-410a-bbd1-e77532149759-kube-api-access-qlmrc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:19.792238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:19.792204    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:20.429057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.429025    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7"
Oct 30 07:00:20.429480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.429022    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvbb7" event={"ID":"529b7749-5afe-410a-bbd1-e77532149759","Type":"ContainerDied","Data":"fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f"}
Oct 30 07:00:20.429480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.429177    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd48168df25bf3ad3c008f9153c9e043334377b4fda53cdc13b42d09dfaab59f"
Oct 30 07:00:20.550765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.550737    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:20.665624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.665589    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\") pod \"72674139-4de1-4ee5-af29-d53a169b27c0\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") "
Oct 30 07:00:20.665817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.665686    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\") pod \"72674139-4de1-4ee5-af29-d53a169b27c0\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") "
Oct 30 07:00:20.665817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.665709    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\" (UniqueName: \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\") pod \"72674139-4de1-4ee5-af29-d53a169b27c0\" (UID: \"72674139-4de1-4ee5-af29-d53a169b27c0\") "
Oct 30 07:00:20.666342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.666288    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle" pod "72674139-4de1-4ee5-af29-d53a169b27c0" (UID: "72674139-4de1-4ee5-af29-d53a169b27c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:20.667740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.667717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq" pod "72674139-4de1-4ee5-af29-d53a169b27c0" (UID: "72674139-4de1-4ee5-af29-d53a169b27c0"). InnerVolumeSpecName "kube-api-access-dmdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:20.669581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.669555    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util" pod "72674139-4de1-4ee5-af29-d53a169b27c0" (UID: "72674139-4de1-4ee5-af29-d53a169b27c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:20.767140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.767053    6738 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-bundle\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:20.767140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.767090    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdwq\" (UniqueName: \"kubernetes.io/projected/72674139-4de1-4ee5-af29-d53a169b27c0-kube-api-access-dmdwq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:20.767140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:20.767103    6738 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72674139-4de1-4ee5-af29-d53a169b27c0-util\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:00:21.434738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:21.434711    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f"
Oct 30 07:00:21.435101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:21.434714    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnjd8f" event={"ID":"72674139-4de1-4ee5-af29-d53a169b27c0","Type":"ContainerDied","Data":"7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e"}
Oct 30 07:00:21.435101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:21.434828    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e33796ed5ccf0690b21d05a9e8f661c5e73c7f31e0b9e4856fef1cc1d96584e"
Oct 30 07:00:27.153830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:27.153790    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:33.751623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:33.751524    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:36.676918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:36.676880    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:36.817056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:36.817015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:37.339880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.339839    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:37.517655 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:00:37.517623    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:00:37.519532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519508    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"]
Oct 30 07:00:37.519806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519794    6738 cpu_manager.go:414] "containerMap: removing container" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="util"
Oct 30 07:00:37.519850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519807    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="util"
Oct 30 07:00:37.519850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519826    6738 cpu_manager.go:414] "containerMap: removing container" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="util"
Oct 30 07:00:37.519850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519832    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="util"
Oct 30 07:00:37.519850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519839    6738 cpu_manager.go:414] "containerMap: removing container" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="extract"
Oct 30 07:00:37.519850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519844    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="extract"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519855    6738 cpu_manager.go:414] "containerMap: removing container" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="extract"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519860    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="extract"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519866    6738 cpu_manager.go:414] "containerMap: removing container" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="pull"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519870    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="pull"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519889    6738 cpu_manager.go:414] "containerMap: removing container" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="pull"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519893    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="pull"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519937    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="72674139-4de1-4ee5-af29-d53a169b27c0" containerName="extract"
Oct 30 07:00:37.519996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.519945    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="529b7749-5afe-410a-bbd1-e77532149759" containerName="extract"
Oct 30 07:00:37.522081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.522066    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.554952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.554924    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"]
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.596790    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.596921    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.596929    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.596955    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.596971    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-q568p\""
Oct 30 07:00:37.597209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.597066    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 07:00:37.638504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.638474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:37.703398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703361    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.703398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703399    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmztq\" (UniqueName: \"kubernetes.io/projected/82621905-a436-4c7a-9bca-0e08d1dd8e5a-kube-api-access-nmztq\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.703842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703486    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.703842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703587    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.703842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703621    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.703842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.703636    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.798806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.798766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:37.804815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804784    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.804883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804827    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.804883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804844    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.804883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804876    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.805023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804901    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmztq\" (UniqueName: \"kubernetes.io/projected/82621905-a436-4c7a-9bca-0e08d1dd8e5a-kube-api-access-nmztq\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.805023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.804943    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.805375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.805355    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.807463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.807440    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82621905-a436-4c7a-9bca-0e08d1dd8e5a-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.807582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.807560    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.807693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.807676    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.807737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.807677    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82621905-a436-4c7a-9bca-0e08d1dd8e5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.829242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.829212    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmztq\" (UniqueName: \"kubernetes.io/projected/82621905-a436-4c7a-9bca-0e08d1dd8e5a-kube-api-access-nmztq\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-hn4ct\" (UID: \"82621905-a436-4c7a-9bca-0e08d1dd8e5a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.835039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.835016    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:37.924557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:37.924522    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:38.183649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:38.183626    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"]
Oct 30 07:00:38.186014 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:00:38.185985    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82621905_a436_4c7a_9bca_0e08d1dd8e5a.slice/crio-7077bf95156500385bacd734b59ed85d5f46fd14e5a5e246692559edc1de936a WatchSource:0}: Error finding container 7077bf95156500385bacd734b59ed85d5f46fd14e5a5e246692559edc1de936a: Status 404 returned error can't find the container with id 7077bf95156500385bacd734b59ed85d5f46fd14e5a5e246692559edc1de936a
Oct 30 07:00:38.498918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:38.498830    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct" event={"ID":"82621905-a436-4c7a-9bca-0e08d1dd8e5a","Type":"ContainerStarted","Data":"7077bf95156500385bacd734b59ed85d5f46fd14e5a5e246692559edc1de936a"}
Oct 30 07:00:38.740451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:38.740411    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:40.118333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:40.118277    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:40.285723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:40.285681    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:40.681789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:40.681651    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:40.811821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:40.811785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:41.516030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:41.515998    6738 generic.go:356] "Generic (PLEG): container finished" podID="82621905-a436-4c7a-9bca-0e08d1dd8e5a" containerID="6470e86bf3936b254ffdc9249ed9b84ba0fd4236acd96056062e1e8ee4cbab85" exitCode=0
Oct 30 07:00:41.516441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:41.516062    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct" event={"ID":"82621905-a436-4c7a-9bca-0e08d1dd8e5a","Type":"ContainerDied","Data":"6470e86bf3936b254ffdc9249ed9b84ba0fd4236acd96056062e1e8ee4cbab85"}
Oct 30 07:00:42.521924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:42.521882    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct" event={"ID":"82621905-a436-4c7a-9bca-0e08d1dd8e5a","Type":"ContainerStarted","Data":"60fea8abc25e58b2a7326c750bd0a4dab6fa01d8881d87b58963ca9e6c486b6d"}
Oct 30 07:00:42.522369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:42.522023    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:42.622248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:42.622197    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct" podStartSLOduration=2.092244952 podStartE2EDuration="5.622183687s" totalImagesPullingTime="3.061120542s" totalInitContainerRuntime="468.818193ms" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:42.608879729 +0000 UTC m=+203.041643618" watchObservedRunningTime="2026-10-30 07:00:42.622183687 +0000 UTC m=+203.054947579"
Oct 30 07:00:46.676732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:46.676700    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:46.807983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:46.807948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:49.913889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:49.913851    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:53.260180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:53.260146    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:53.527718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:53.527639    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-hn4ct"
Oct 30 07:00:53.671102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:53.671059    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:54.669061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:54.669031    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:55.450076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:55.450042    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:00:59.106585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:00:59.106545    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:00.780234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:00.780195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:00.928427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:00.928392    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:01.122074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:01.122030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:01.250859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:01.250823    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:01.906266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:01.906236    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:02.047465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:02.047429    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:03.480562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:03.480526    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:03.852041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:03.851951    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:04.245252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:04.245211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:05.061159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:05.061120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:05.524889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:05.524858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:07.517581 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:01:07.517543    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:01:07.519003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:07.517890    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:07.693353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:07.693319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:09.465705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:09.465660    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:09.592521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:09.592487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:11.440636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.440601    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:11.663354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.663319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:11.775997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.775909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:11.818489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.818456    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-9b295"]
Oct 30 07:01:11.820766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.820748    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.854386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.854352    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 07:01:11.854386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.854352    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-v57gj\""
Oct 30 07:01:11.854586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.854476    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 07:01:11.855850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.855831    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9b295"]
Oct 30 07:01:11.874795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.874761    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-bound-sa-token\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.874963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.874813    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cph\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-kube-api-access-p7cph\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.975361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.975286    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-bound-sa-token\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.975567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.975403    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cph\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-kube-api-access-p7cph\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.999213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.999178    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cph\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-kube-api-access-p7cph\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:11.999408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:11.999331    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d23149d7-72d3-40e1-8d55-4cea8b3703f7-bound-sa-token\") pod \"cert-manager-597b95cb84-9b295\" (UID: \"d23149d7-72d3-40e1-8d55-4cea8b3703f7\") " pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:12.143796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:12.143762    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9b295"
Oct 30 07:01:12.461931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:12.461897    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9b295"]
Oct 30 07:01:12.467932 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:01:12.464348    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd23149d7_72d3_40e1_8d55_4cea8b3703f7.slice/crio-76272f9f00b1ad4cac548f811b231f4917208d9bed95c07cab4adc127cef64b9 WatchSource:0}: Error finding container 76272f9f00b1ad4cac548f811b231f4917208d9bed95c07cab4adc127cef64b9: Status 404 returned error can't find the container with id 76272f9f00b1ad4cac548f811b231f4917208d9bed95c07cab4adc127cef64b9
Oct 30 07:01:12.639283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:12.639243    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9b295" event={"ID":"d23149d7-72d3-40e1-8d55-4cea8b3703f7","Type":"ContainerStarted","Data":"76272f9f00b1ad4cac548f811b231f4917208d9bed95c07cab4adc127cef64b9"}
Oct 30 07:01:12.826864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:12.826781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:12.992757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:12.992724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.116430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.116349    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.261057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.261018    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.642721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.642681    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.654642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.654613    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"]
Oct 30 07:01:13.657399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.657373    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.678269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.678231    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"]
Oct 30 07:01:13.690361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.690333    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 07:01:13.690725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.690599    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 07:01:13.690725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.690699    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vg4t5\""
Oct 30 07:01:13.691376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.691355    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039c128-a834-422d-8dbe-5b1bda441bc6-tmp\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.691479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.691410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ckh\" (UniqueName: \"kubernetes.io/projected/3039c128-a834-422d-8dbe-5b1bda441bc6-kube-api-access-b8ckh\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.691687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.691545    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3039c128-a834-422d-8dbe-5b1bda441bc6-tls-certs\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.792342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.792283    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039c128-a834-422d-8dbe-5b1bda441bc6-tmp\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.792342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.792345    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ckh\" (UniqueName: \"kubernetes.io/projected/3039c128-a834-422d-8dbe-5b1bda441bc6-kube-api-access-b8ckh\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.792599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.792394    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3039c128-a834-422d-8dbe-5b1bda441bc6-tls-certs\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.794805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.794777    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039c128-a834-422d-8dbe-5b1bda441bc6-tmp\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.795114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.795094    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3039c128-a834-422d-8dbe-5b1bda441bc6-tls-certs\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.820933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.820889    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ckh\" (UniqueName: \"kubernetes.io/projected/3039c128-a834-422d-8dbe-5b1bda441bc6-kube-api-access-b8ckh\") pod \"kube-auth-proxy-54599bd898-4gdnn\" (UID: \"3039c128-a834-422d-8dbe-5b1bda441bc6\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:13.822086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.821976    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.935285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.935189    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:13.973544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:13.973509    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"
Oct 30 07:01:14.062194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:14.062147    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:14.287784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:14.287372    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-4gdnn"]
Oct 30 07:01:14.289418 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:01:14.289392    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3039c128_a834_422d_8dbe_5b1bda441bc6.slice/crio-8363e681fc78c90922c4c3ce3579cddcfcbda716725d6a2ed4d27e134f37623e WatchSource:0}: Error finding container 8363e681fc78c90922c4c3ce3579cddcfcbda716725d6a2ed4d27e134f37623e: Status 404 returned error can't find the container with id 8363e681fc78c90922c4c3ce3579cddcfcbda716725d6a2ed4d27e134f37623e
Oct 30 07:01:14.651580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:14.651539    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn" event={"ID":"3039c128-a834-422d-8dbe-5b1bda441bc6","Type":"ContainerStarted","Data":"8363e681fc78c90922c4c3ce3579cddcfcbda716725d6a2ed4d27e134f37623e"}
Oct 30 07:01:15.233481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:15.233445    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:15.658851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:15.658805    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9b295" event={"ID":"d23149d7-72d3-40e1-8d55-4cea8b3703f7","Type":"ContainerStarted","Data":"25096a42e6949095642318af49316636c0da4d47358e7dd103f5ffa70ded6142"}
Oct 30 07:01:15.700025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:15.699967    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-9b295" podStartSLOduration=1.871703774 podStartE2EDuration="4.699949817s" totalImagesPullingTime="2.828246043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:15.699222715 +0000 UTC m=+236.131986609" watchObservedRunningTime="2026-10-30 07:01:15.699949817 +0000 UTC m=+236.132713726"
Oct 30 07:01:16.458677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:16.458632    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:16.616814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:16.616775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:17.078051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.077962    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:17.191179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.191145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:17.668869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.668831    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn" event={"ID":"3039c128-a834-422d-8dbe-5b1bda441bc6","Type":"ContainerStarted","Data":"91a368404cc76e9dbc551e8688f305a5b9a027daa51cb44e3649bba7aa181850"}
Oct 30 07:01:17.722633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.722568    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54599bd898-4gdnn" podStartSLOduration=2.291590783 podStartE2EDuration="4.722550019s" totalImagesPullingTime="2.430959236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:17.721883294 +0000 UTC m=+238.154647187" watchObservedRunningTime="2026-10-30 07:01:17.722550019 +0000 UTC m=+238.155313916"
Oct 30 07:01:17.723638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.723612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:17.850412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:17.850380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:22.801363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.801321    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"]
Oct 30 07:01:22.807051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.807026    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.836701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.836669    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4fzt5\""
Oct 30 07:01:22.836941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.836720    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 07:01:22.836941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.836745    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 07:01:22.863800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.863763    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 07:01:22.864459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.864435    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"]
Oct 30 07:01:22.878908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.878872    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.879038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.878923    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wzg\" (UniqueName: \"kubernetes.io/projected/cff3c19d-9829-428c-a1c0-a5b623913630-kube-api-access-d8wzg\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.879038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.879014    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.879111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.879037    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cff3c19d-9829-428c-a1c0-a5b623913630-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.979989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.979950    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.980159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.980010    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wzg\" (UniqueName: \"kubernetes.io/projected/cff3c19d-9829-428c-a1c0-a5b623913630-kube-api-access-d8wzg\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.980159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.980057    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.980159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.980081    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cff3c19d-9829-428c-a1c0-a5b623913630-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.980454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.980435    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.980513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.980490    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff3c19d-9829-428c-a1c0-a5b623913630-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:22.981073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:22.981056    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cff3c19d-9829-428c-a1c0-a5b623913630-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:23.004397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:23.004368    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wzg\" (UniqueName: \"kubernetes.io/projected/cff3c19d-9829-428c-a1c0-a5b623913630-kube-api-access-d8wzg\") pod \"kserve-module-controller-manager-7566cf5fc8-v4kpg\" (UID: \"cff3c19d-9829-428c-a1c0-a5b623913630\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:23.120354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:23.120239    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:23.441789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:23.441764    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"]
Oct 30 07:01:23.443239 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:01:23.443207    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff3c19d_9829_428c_a1c0_a5b623913630.slice/crio-494c9399c6c03358777b537159b7042b0782d2143cdc0304242f16c717fc1ade WatchSource:0}: Error finding container 494c9399c6c03358777b537159b7042b0782d2143cdc0304242f16c717fc1ade: Status 404 returned error can't find the container with id 494c9399c6c03358777b537159b7042b0782d2143cdc0304242f16c717fc1ade
Oct 30 07:01:23.694223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:23.694143    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg" event={"ID":"cff3c19d-9829-428c-a1c0-a5b623913630","Type":"ContainerStarted","Data":"494c9399c6c03358777b537159b7042b0782d2143cdc0304242f16c717fc1ade"}
Oct 30 07:01:24.477478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:24.477437    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:24.596154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:24.596120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:25.704267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:25.704229    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg" event={"ID":"cff3c19d-9829-428c-a1c0-a5b623913630","Type":"ContainerStarted","Data":"a9de1e3ced33e1ee3a13497aef085789b85a6dc6694dba30f12039e4dc37b578"}
Oct 30 07:01:25.704774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:25.704317    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:25.749480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:25.749430    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg" podStartSLOduration=2.190704503 podStartE2EDuration="3.749416496s" totalImagesPullingTime="1.558711993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:25.74820923 +0000 UTC m=+246.180973128" watchObservedRunningTime="2026-10-30 07:01:25.749416496 +0000 UTC m=+246.182180438"
Oct 30 07:01:27.279739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:27.279693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:30.092340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:30.092277    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:30.204044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:30.204009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:32.611231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:32.611194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:32.723663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:32.723627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:35.890215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:35.890183    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:36.009579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:36.009542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:36.121454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:36.121413    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:37.516829 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:01:37.516794    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:01:42.509046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:42.509013    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:42.617703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:42.617661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:46.709338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:46.709284    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4kpg"
Oct 30 07:01:49.049373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:49.049336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:49.162421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:49.162389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:50.168997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:50.168966    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:51.671338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:51.671279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:51.796986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:51.796955    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:54.970080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:54.970040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:55.101700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:55.101662    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:55.531153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:55.531116    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:55.648644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:55.648598    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:55.766110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:55.766078    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:55.893290 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:55.893255    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:01:56.011640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:01:56.011595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:01.430529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:01.430443    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:01.552008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:01.551968    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:02.295279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:02.295233    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:02.419176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:02.419143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:07.517053 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:07.517011    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:02:08.420569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:08.420531    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:08.532153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:08.532094    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:08.895247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:08.895210    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:09.014671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:09.014634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:17.210742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.210700    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"]
Oct 30 07:02:17.216697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.216663    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.249998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.249958    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"]
Oct 30 07:02:17.383351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383280    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383408    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75h8\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-kube-api-access-m75h8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383484    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383509    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383527    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383624    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5dd99-6942-4130-b01b-00413a307288-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.383672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.383660    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484617    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484657    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484691    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75h8\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-kube-api-access-m75h8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484734    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484762    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484783    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.484980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.484872    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5dd99-6942-4130-b01b-00413a307288-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.485572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.485542    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.487658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.487633    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.487757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.487652    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b5dd99-6942-4130-b01b-00413a307288-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.487835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.487817    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.487872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.487817    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88b5dd99-6942-4130-b01b-00413a307288-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.509369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.509333    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.509538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.509502    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75h8\" (UniqueName: \"kubernetes.io/projected/88b5dd99-6942-4130-b01b-00413a307288-kube-api-access-m75h8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lpfqr\" (UID: \"88b5dd99-6942-4130-b01b-00413a307288\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.529198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.529161    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.851892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.851862    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"]
Oct 30 07:02:17.853730 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:02:17.853705    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b5dd99_6942_4130_b01b_00413a307288.slice/crio-d819695f6bba01558e17c9b64124d994a6ed6393e96e58b5cb446c5695e2d024 WatchSource:0}: Error finding container d819695f6bba01558e17c9b64124d994a6ed6393e96e58b5cb446c5695e2d024: Status 404 returned error can't find the container with id d819695f6bba01558e17c9b64124d994a6ed6393e96e58b5cb446c5695e2d024
Oct 30 07:02:17.855727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.855694    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:02:17.855800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.855761    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:02:17.922863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.922822    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr" event={"ID":"88b5dd99-6942-4130-b01b-00413a307288","Type":"ContainerStarted","Data":"bdb4332c5ab9c31b8bef81886c73ed6b55209051e1c76a1d414d02dbb143a659"}
Oct 30 07:02:17.923131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.923112    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:17.923258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.923241    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr" event={"ID":"88b5dd99-6942-4130-b01b-00413a307288","Type":"ContainerStarted","Data":"d819695f6bba01558e17c9b64124d994a6ed6393e96e58b5cb446c5695e2d024"}
Oct 30 07:02:17.973223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:17.973154    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr" podStartSLOduration=0.973132852 podStartE2EDuration="973.132852ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:17.972950829 +0000 UTC m=+298.405714722" watchObservedRunningTime="2026-10-30 07:02:17.973132852 +0000 UTC m=+298.405896748"
Oct 30 07:02:18.404882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.404833    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:18.522970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.522931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:18.639613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.639575    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:18.773560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.773454    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:18.910599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.910564    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:18.931969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.931932    6738 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lpfqr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:02:18.932161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:18.932004    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr" podUID="88b5dd99-6942-4130-b01b-00413a307288" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:02:19.244176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:19.244142    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:19.934216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:19.934189    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lpfqr"
Oct 30 07:02:20.024988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.024958    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:02:20.025178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.025157    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerName="discovery" containerID="cri-o://a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996" gracePeriod=30
Oct 30 07:02:20.262941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.262911    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:02:20.312138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312107    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312140    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312196    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312234    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312415    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312469    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.312514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.312496    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\") pod \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\" (UID: \"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff\") "
Oct 30 07:02:20.313110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.313065    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:02:20.315179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.315153    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "kube-api-access-6sl9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:02:20.315285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.315184    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:02:20.315379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.315327    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:02:20.315861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.315839    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:02:20.315945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.315887    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:02:20.316106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.316082    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs" pod "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" (UID: "a9e1ac56-e0e1-45fa-84c7-f5ac047027ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:02:20.413608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413567    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sl9t\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-kube-api-access-6sl9t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413602    6738 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-local-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413613    6738 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-dns-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413622    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413631    6738 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-kubeconfig\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413639    6738 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-cacerts\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.413842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.413647    6738 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff-istio-csr-ca-configmap\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:20.939603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.939567    6738 generic.go:356] "Generic (PLEG): container finished" podID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerID="a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996" exitCode=0
Oct 30 07:02:20.940073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.939613    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"
Oct 30 07:02:20.940073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.939665    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" event={"ID":"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff","Type":"ContainerDied","Data":"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"}
Oct 30 07:02:20.940073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.939713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q" event={"ID":"a9e1ac56-e0e1-45fa-84c7-f5ac047027ff","Type":"ContainerDied","Data":"335c1e25e623cd85c3fccd15781a75e6afd9cd9987dcd438988957d04ff47737"}
Oct 30 07:02:20.940073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.939739    6738 scope.go:122] "RemoveContainer" containerID="a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"
Oct 30 07:02:20.954275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.954248    6738 scope.go:122] "RemoveContainer" containerID="a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"
Oct 30 07:02:20.954618 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:20.954597    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996\": container with ID starting with a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996 not found: ID does not exist" containerID="a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"
Oct 30 07:02:20.954710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:20.954625    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996"} err="failed to get container status \"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996\": rpc error: code = NotFound desc = could not find container \"a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996\": container with ID starting with a6ab95587526119e951e4b3a290175b221950d3871d3371396ca02a28efea996 not found: ID does not exist"
Oct 30 07:02:21.012461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.012424    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:02:21.025156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.025124    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lvh8q"]
Oct 30 07:02:21.447143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.447106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:21.579469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.579430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:21.688736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.688696    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" path="/var/lib/kubelet/pods/a9e1ac56-e0e1-45fa-84c7-f5ac047027ff/volumes"
Oct 30 07:02:21.723906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.723817    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:21.917077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:21.917046    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:22.051906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.051807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:22.147888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.147852    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:22.148332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.148314    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerName="discovery"
Oct 30 07:02:22.148332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.148332    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerName="discovery"
Oct 30 07:02:22.148450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.148390    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e1ac56-e0e1-45fa-84c7-f5ac047027ff" containerName="discovery"
Oct 30 07:02:22.152903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.152874    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.168845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.168814    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 07:02:22.168845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.168813    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 07:02:22.169042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.168886    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rdtrb\""
Oct 30 07:02:22.171933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.171907    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 07:02:22.174063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.174039    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:22.216711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.216668    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:22.230965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.230927    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.230965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.230965    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2np\" (UniqueName: \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.231179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.231063    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.332021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.331935    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.332021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.331989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.332021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.332020    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2np\" (UniqueName: \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.334617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.334585    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.334752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.334630    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.356447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.356414    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2np\" (UniqueName: \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\") pod \"maas-controller-fcfd895c4-77bx5\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") " pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.376937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.376898    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:22.468893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.468851    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:22.651502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.651464    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:22.747089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.747055    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7df767b8fd-g9dgt"]
Oct 30 07:02:22.751419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.751392    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.769624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.769599    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:22.771510 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:02:22.771484    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175be214_96bb_4319_ade4_45f1942d2d67.slice/crio-f8a6043bd7b0f7c8848cddda5e1a9e3a6e632d027b1cc5c33e041110b50c622f WatchSource:0}: Error finding container f8a6043bd7b0f7c8848cddda5e1a9e3a6e632d027b1cc5c33e041110b50c622f: Status 404 returned error can't find the container with id f8a6043bd7b0f7c8848cddda5e1a9e3a6e632d027b1cc5c33e041110b50c622f
Oct 30 07:02:22.772819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.772799    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7df767b8fd-g9dgt"]
Oct 30 07:02:22.835695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.835656    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-metrics-certs\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.835882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.835768    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsm7\" (UniqueName: \"kubernetes.io/projected/b0c9cc41-c7e2-498c-812f-93268565f166-kube-api-access-rdsm7\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.835882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.835836    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-webhook-cert\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.937039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.936948    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-metrics-certs\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.937039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.937015    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsm7\" (UniqueName: \"kubernetes.io/projected/b0c9cc41-c7e2-498c-812f-93268565f166-kube-api-access-rdsm7\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.937039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.937035    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-webhook-cert\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.939585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.939555    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-webhook-cert\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.939585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.939575    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0c9cc41-c7e2-498c-812f-93268565f166-metrics-certs\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:22.949897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.949859    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-77bx5" event={"ID":"175be214-96bb-4319-ade4-45f1942d2d67","Type":"ContainerStarted","Data":"f8a6043bd7b0f7c8848cddda5e1a9e3a6e632d027b1cc5c33e041110b50c622f"}
Oct 30 07:02:22.972524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:22.972496    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsm7\" (UniqueName: \"kubernetes.io/projected/b0c9cc41-c7e2-498c-812f-93268565f166-kube-api-access-rdsm7\") pod \"maas-controller-7df767b8fd-g9dgt\" (UID: \"b0c9cc41-c7e2-498c-812f-93268565f166\") " pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:23.064595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:23.064558    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:23.374362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:23.374331    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7df767b8fd-g9dgt"]
Oct 30 07:02:23.376865 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:02:23.376821    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c9cc41_c7e2_498c_812f_93268565f166.slice/crio-0c50fedbdb29cde154648da4c50a2c9185b8d51c26cd8d69f9286dea99a997b1 WatchSource:0}: Error finding container 0c50fedbdb29cde154648da4c50a2c9185b8d51c26cd8d69f9286dea99a997b1: Status 404 returned error can't find the container with id 0c50fedbdb29cde154648da4c50a2c9185b8d51c26cd8d69f9286dea99a997b1
Oct 30 07:02:23.960445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:23.960356    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7df767b8fd-g9dgt" event={"ID":"b0c9cc41-c7e2-498c-812f-93268565f166","Type":"ContainerStarted","Data":"0c50fedbdb29cde154648da4c50a2c9185b8d51c26cd8d69f9286dea99a997b1"}
Oct 30 07:02:25.778614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.778527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:25.972092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.972053    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-77bx5" event={"ID":"175be214-96bb-4319-ade4-45f1942d2d67","Type":"ContainerStarted","Data":"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"}
Oct 30 07:02:25.972304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.972227    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:25.973520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.973496    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7df767b8fd-g9dgt" event={"ID":"b0c9cc41-c7e2-498c-812f-93268565f166","Type":"ContainerStarted","Data":"efeb41ae4cd0d7363a34f096ac5215d9caa63e06d0ca7fef7b58d860c50df986"}
Oct 30 07:02:25.973657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.973639    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:25.979709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:25.979685    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:26.049923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:26.049813    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-fcfd895c4-77bx5" podStartSLOduration=1.310785713 podStartE2EDuration="4.04979728s" totalImagesPullingTime="2.739011567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:26.046711134 +0000 UTC m=+306.479475027" watchObservedRunningTime="2026-10-30 07:02:26.04979728 +0000 UTC m=+306.482561172"
Oct 30 07:02:27.394251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.394195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:27.833055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.832929    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7df767b8fd-g9dgt" podStartSLOduration=3.660587673 podStartE2EDuration="5.832907825s" totalImagesPullingTime="2.172320152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:26.129186879 +0000 UTC m=+306.561950774" watchObservedRunningTime="2026-10-30 07:02:27.832907825 +0000 UTC m=+308.265671718"
Oct 30 07:02:27.833647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.833625    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"]
Oct 30 07:02:27.837137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.837122    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.871498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.871469    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qh2r4\""
Oct 30 07:02:27.871689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.871602    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 07:02:27.871689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.871657    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 07:02:27.871865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.871697    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 07:02:27.873007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.872978    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"]
Oct 30 07:02:27.882166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.882136    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.882315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.882182    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.882315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.882265    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/afbea8a0-a4f1-489c-97f0-25b42a9e9239-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.882414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.882389    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt4m\" (UniqueName: \"kubernetes.io/projected/afbea8a0-a4f1-489c-97f0-25b42a9e9239-kube-api-access-gvt4m\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.928553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.928514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:27.982808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.982780    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.983018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.982817    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/afbea8a0-a4f1-489c-97f0-25b42a9e9239-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.983018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.982851    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt4m\" (UniqueName: \"kubernetes.io/projected/afbea8a0-a4f1-489c-97f0-25b42a9e9239-kube-api-access-gvt4m\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.983018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.982902    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.983018 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:27.982961    6738 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 07:02:27.983018 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:27.983022    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source podName:afbea8a0-a4f1-489c-97f0-25b42a9e9239 nodeName:}" failed. No retries permitted until 2026-10-30 07:02:28.483003468 +0000 UTC m=+308.915767352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-6chbj" (UID: "afbea8a0-a4f1-489c-97f0-25b42a9e9239") : configmap "usage-tenancy-proxy-source" not found
Oct 30 07:02:27.983595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.983577    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:27.985319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:27.985269    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/afbea8a0-a4f1-489c-97f0-25b42a9e9239-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:28.007258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:28.007219    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt4m\" (UniqueName: \"kubernetes.io/projected/afbea8a0-a4f1-489c-97f0-25b42a9e9239-kube-api-access-gvt4m\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:28.047348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:28.047307    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:28.485856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:28.485808    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:28.486645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:28.486624    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/afbea8a0-a4f1-489c-97f0-25b42a9e9239-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6chbj\" (UID: \"afbea8a0-a4f1-489c-97f0-25b42a9e9239\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:28.749952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:28.749859    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:29.049649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:29.049617    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"]
Oct 30 07:02:29.051162 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:02:29.051129    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbea8a0_a4f1_489c_97f0_25b42a9e9239.slice/crio-44c3e26a76ada2784668126ac8492c450615db3f7bfdb1af5b4c2c4cc83ac6bf WatchSource:0}: Error finding container 44c3e26a76ada2784668126ac8492c450615db3f7bfdb1af5b4c2c4cc83ac6bf: Status 404 returned error can't find the container with id 44c3e26a76ada2784668126ac8492c450615db3f7bfdb1af5b4c2c4cc83ac6bf
Oct 30 07:02:29.991241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:29.991199    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj" event={"ID":"afbea8a0-a4f1-489c-97f0-25b42a9e9239","Type":"ContainerStarted","Data":"44c3e26a76ada2784668126ac8492c450615db3f7bfdb1af5b4c2c4cc83ac6bf"}
Oct 30 07:02:31.573914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:31.573875    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:31.698480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:31.698437    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:32.209958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.209912    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"]
Oct 30 07:02:32.213912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.213880    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.252517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252480    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 07:02:32.252690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252626    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 07:02:32.252764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252693    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 07:02:32.252764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252762    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 07:02:32.252879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252861    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 07:02:32.252879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.252484    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-fcbld\""
Oct 30 07:02:32.253960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.253932    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"]
Oct 30 07:02:32.325361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.325316    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-metrics-certs\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.325557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.325371    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bq4\" (UniqueName: \"kubernetes.io/projected/2e71e6c3-d28e-4885-a230-4a40113490e5-kube-api-access-h2bq4\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.325557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.325500    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-cert\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.325676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.325557    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e71e6c3-d28e-4885-a230-4a40113490e5-manager-config\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.426698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.426659    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-metrics-certs\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.426698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.426710    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bq4\" (UniqueName: \"kubernetes.io/projected/2e71e6c3-d28e-4885-a230-4a40113490e5-kube-api-access-h2bq4\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.426963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.426743    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-cert\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.426963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.426770    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e71e6c3-d28e-4885-a230-4a40113490e5-manager-config\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.427541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.427498    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e71e6c3-d28e-4885-a230-4a40113490e5-manager-config\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.430043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.429980    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-metrics-certs\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.430043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.429980    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e71e6c3-d28e-4885-a230-4a40113490e5-cert\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.452516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.452477    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bq4\" (UniqueName: \"kubernetes.io/projected/2e71e6c3-d28e-4885-a230-4a40113490e5-kube-api-access-h2bq4\") pod \"jobset-controller-manager-5cbf5ddb54-dwht5\" (UID: \"2e71e6c3-d28e-4885-a230-4a40113490e5\") " pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.530347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.530245    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:32.852220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:32.852190    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"]
Oct 30 07:02:32.854469 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:02:32.854436    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e71e6c3_d28e_4885_a230_4a40113490e5.slice/crio-e5bfd80e60ad4b17c16ecff267006366e23bcadf48c386f50431ef001116e323 WatchSource:0}: Error finding container e5bfd80e60ad4b17c16ecff267006366e23bcadf48c386f50431ef001116e323: Status 404 returned error can't find the container with id e5bfd80e60ad4b17c16ecff267006366e23bcadf48c386f50431ef001116e323
Oct 30 07:02:33.012120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:33.012077    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5" event={"ID":"2e71e6c3-d28e-4885-a230-4a40113490e5","Type":"ContainerStarted","Data":"e5bfd80e60ad4b17c16ecff267006366e23bcadf48c386f50431ef001116e323"}
Oct 30 07:02:36.981362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:36.981330    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7df767b8fd-g9dgt"
Oct 30 07:02:36.981823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:36.981416    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:37.035704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.035663    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5" event={"ID":"2e71e6c3-d28e-4885-a230-4a40113490e5","Type":"ContainerStarted","Data":"db582fbcb304a9196c03233f5bbd4737e98ad0e9b85b3fe0ea699b4eb75f72e8"}
Oct 30 07:02:37.036019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.035995    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:37.055694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.055663    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:37.056180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.055899    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-fcfd895c4-77bx5" podUID="175be214-96bb-4319-ade4-45f1942d2d67" containerName="manager" containerID="cri-o://605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007" gracePeriod=30
Oct 30 07:02:37.113648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.113467    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5" podStartSLOduration=1.888120139 podStartE2EDuration="5.113449209s" totalImagesPullingTime="3.22532907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:37.112879955 +0000 UTC m=+317.545643849" watchObservedRunningTime="2026-10-30 07:02:37.113449209 +0000 UTC m=+317.546213098"
Oct 30 07:02:37.314807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.314773    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:37.395200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.395162    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\") pod \"175be214-96bb-4319-ade4-45f1942d2d67\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") "
Oct 30 07:02:37.395401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.395220    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\" (UniqueName: \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\") pod \"175be214-96bb-4319-ade4-45f1942d2d67\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") "
Oct 30 07:02:37.395401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.395346    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\") pod \"175be214-96bb-4319-ade4-45f1942d2d67\" (UID: \"175be214-96bb-4319-ade4-45f1942d2d67\") "
Oct 30 07:02:37.398110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.398071    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs" pod "175be214-96bb-4319-ade4-45f1942d2d67" (UID: "175be214-96bb-4319-ade4-45f1942d2d67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:02:37.398263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.398104    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert" pod "175be214-96bb-4319-ade4-45f1942d2d67" (UID: "175be214-96bb-4319-ade4-45f1942d2d67"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:02:37.398263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.398107    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np" pod "175be214-96bb-4319-ade4-45f1942d2d67" (UID: "175be214-96bb-4319-ade4-45f1942d2d67"). InnerVolumeSpecName "kube-api-access-5t2np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:02:37.496816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.496783    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:37.496816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.496816    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2np\" (UniqueName: \"kubernetes.io/projected/175be214-96bb-4319-ade4-45f1942d2d67-kube-api-access-5t2np\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:37.497038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:37.496832    6738 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/175be214-96bb-4319-ade4-45f1942d2d67-webhook-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:02:37.520526 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:37.520488    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:02:38.042400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.042289    6738 generic.go:356] "Generic (PLEG): container finished" podID="175be214-96bb-4319-ade4-45f1942d2d67" containerID="605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007" exitCode=0
Oct 30 07:02:38.042400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.042374    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-77bx5"
Oct 30 07:02:38.042874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.042390    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-77bx5" event={"ID":"175be214-96bb-4319-ade4-45f1942d2d67","Type":"ContainerDied","Data":"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"}
Oct 30 07:02:38.042874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.042444    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-77bx5" event={"ID":"175be214-96bb-4319-ade4-45f1942d2d67","Type":"ContainerDied","Data":"f8a6043bd7b0f7c8848cddda5e1a9e3a6e632d027b1cc5c33e041110b50c622f"}
Oct 30 07:02:38.042874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.042467    6738 scope.go:122] "RemoveContainer" containerID="605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"
Oct 30 07:02:38.052678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.052650    6738 scope.go:122] "RemoveContainer" containerID="605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"
Oct 30 07:02:38.053038 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:02:38.053015    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007\": container with ID starting with 605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007 not found: ID does not exist" containerID="605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"
Oct 30 07:02:38.053129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.053052    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007"} err="failed to get container status \"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007\": rpc error: code = NotFound desc = could not find container \"605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007\": container with ID starting with 605374e9cd1b83255dcf763f94df10c6dc4ae25d1479cd718f69b7a18f823007 not found: ID does not exist"
Oct 30 07:02:38.095795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.095757    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:38.108811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:38.108771    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-77bx5"]
Oct 30 07:02:39.689188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:39.689153    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175be214-96bb-4319-ade4-45f1942d2d67" path="/var/lib/kubelet/pods/175be214-96bb-4319-ade4-45f1942d2d67/volumes"
Oct 30 07:02:41.059064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:41.059008    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj" event={"ID":"afbea8a0-a4f1-489c-97f0-25b42a9e9239","Type":"ContainerStarted","Data":"dec71291fe57fac6fb924849953aa6ca403d9d4a59d9e736ef770821b1443be3"}
Oct 30 07:02:41.059491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:41.059114    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:41.103268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:41.103208    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj" podStartSLOduration=2.62011314 podStartE2EDuration="14.103191782s" totalImagesPullingTime="11.483078642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:41.101993794 +0000 UTC m=+321.534757688" watchObservedRunningTime="2026-10-30 07:02:41.103191782 +0000 UTC m=+321.535955675"
Oct 30 07:02:48.047526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:48.047492    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5cbf5ddb54-dwht5"
Oct 30 07:02:52.068953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:52.068920    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6chbj"
Oct 30 07:02:57.376771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:57.376736    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:57.507422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:57.507380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:58.900781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:58.900743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:02:59.047885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:02:59.047854    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:03:07.517101 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:03:07.517064    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:03:09.838484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:03:09.838449    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:03:10.004910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:03:10.004876    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:03:27.532037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:03:27.531999    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:03:37.516908 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:03:37.516824    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:03:56.328347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:03:56.328281    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:03:56.446465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:03:56.446425    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:07.518678 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:04:07.518633    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:04:07.802556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:07.802465    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:07.920044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:07.920005    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:12.092838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:12.092788    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:12.209203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:12.209162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:27.651513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:27.651476    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:36.603600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:36.603568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:37.164012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:37.163975    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:37.516225 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:04:37.516141    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:04:38.559164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:38.559127    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:38.684133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:38.684097    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:43.665883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:43.665845    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:48.036964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:48.036933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:50.643413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:50.643376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:52.980428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:52.980393    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"]
Oct 30 07:04:52.980974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:52.980949    6738 cpu_manager.go:414] "containerMap: removing container" podUID="175be214-96bb-4319-ade4-45f1942d2d67" containerName="manager"
Oct 30 07:04:52.980974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:52.980973    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175be214-96bb-4319-ade4-45f1942d2d67" containerName="manager"
Oct 30 07:04:52.981172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:52.981057    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="175be214-96bb-4319-ade4-45f1942d2d67" containerName="manager"
Oct 30 07:04:52.984090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:52.984069    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.002181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.002154    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 07:04:53.014647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.014619    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 07:04:53.014856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.014827    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 07:04:53.014946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.014905    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rlmfx\""
Oct 30 07:04:53.015875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.015856    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"]
Oct 30 07:04:53.045263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.045221    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:53.087745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087715    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.087922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087762    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.087922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087784    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.087922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087888    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5z7h\" (UniqueName: \"kubernetes.io/projected/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-kube-api-access-f5z7h\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.087922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087915    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.088089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.087996    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189189    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189239    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189286    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189325    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189372    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5z7h\" (UniqueName: \"kubernetes.io/projected/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-kube-api-access-f5z7h\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189403    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189737    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189784    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.189836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.189824    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.191601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.191580    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.191737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.191720    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.213375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.213343    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5z7h\" (UniqueName: \"kubernetes.io/projected/a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f-kube-api-access-f5z7h\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-95n5t\" (UID: \"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.300176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.300080    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:53.596568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.596543    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"]
Oct 30 07:04:53.598788 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:04:53.598760    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fb61a3_1a95_4d09_8c10_56fdcafa1c1f.slice/crio-0e08db7e3ef4325ff9ab35e753d0885c40c08cb50724af819abed2e4a239cc94 WatchSource:0}: Error finding container 0e08db7e3ef4325ff9ab35e753d0885c40c08cb50724af819abed2e4a239cc94: Status 404 returned error can't find the container with id 0e08db7e3ef4325ff9ab35e753d0885c40c08cb50724af819abed2e4a239cc94
Oct 30 07:04:53.650946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:53.650911    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t" event={"ID":"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f","Type":"ContainerStarted","Data":"0e08db7e3ef4325ff9ab35e753d0885c40c08cb50724af819abed2e4a239cc94"}
Oct 30 07:04:55.131566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:55.131535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:55.661517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:55.661479    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t" event={"ID":"a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f","Type":"ContainerStarted","Data":"df2a5ce9321d72e2cea39baa1c1819aa7b481537c9766a66f9182575537179aa"}
Oct 30 07:04:55.705560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:55.705510    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t" podStartSLOduration=2.684998385 podStartE2EDuration="3.705494424s" totalImagesPullingTime="1.020496039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:55.704545746 +0000 UTC m=+456.137309639" watchObservedRunningTime="2026-10-30 07:04:55.705494424 +0000 UTC m=+456.138258317"
Oct 30 07:04:56.301145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:56.301113    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:04:56.340348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:56.340313    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:57.752069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:57.752025    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:58.667230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.667199    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"]
Oct 30 07:04:58.670861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.670828    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.686102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.686077    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 07:04:58.699485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.699457    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"]
Oct 30 07:04:58.738342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738288    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.738523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738427    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.738523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2n2h\" (UniqueName: \"kubernetes.io/projected/26c75063-a26f-4073-9561-596ad84493e8-kube-api-access-s2n2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.738523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738520    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.738655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738587    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c75063-a26f-4073-9561-596ad84493e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.738723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.738706    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.743058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.743024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:04:58.839508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839521    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n2h\" (UniqueName: \"kubernetes.io/projected/26c75063-a26f-4073-9561-596ad84493e8-kube-api-access-s2n2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839561    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839580    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c75063-a26f-4073-9561-596ad84493e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839602    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839635    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839913    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.839988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.839962    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.840269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.840028    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.841848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.841830    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c75063-a26f-4073-9561-596ad84493e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.847806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.842107    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c75063-a26f-4073-9561-596ad84493e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.864068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.864034    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n2h\" (UniqueName: \"kubernetes.io/projected/26c75063-a26f-4073-9561-596ad84493e8-kube-api-access-s2n2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn\" (UID: \"26c75063-a26f-4073-9561-596ad84493e8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:58.984593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:58.984509    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:04:59.282011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.281834    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"]
Oct 30 07:04:59.284637 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:04:59.284601    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c75063_a26f_4073_9561_596ad84493e8.slice/crio-7b3b6588979032cfcb5920da2f03be208d382ee7d568aebf88df03179f63c5ff WatchSource:0}: Error finding container 7b3b6588979032cfcb5920da2f03be208d382ee7d568aebf88df03179f63c5ff: Status 404 returned error can't find the container with id 7b3b6588979032cfcb5920da2f03be208d382ee7d568aebf88df03179f63c5ff
Oct 30 07:04:59.680501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.680455    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn" event={"ID":"26c75063-a26f-4073-9561-596ad84493e8","Type":"ContainerStarted","Data":"c036699ca299bd180d9756c3fd03ec3bab99d4a82ab9b5cf8d4dfeace9d6c700"}
Oct 30 07:04:59.680674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.680523    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn" event={"ID":"26c75063-a26f-4073-9561-596ad84493e8","Type":"ContainerStarted","Data":"7b3b6588979032cfcb5920da2f03be208d382ee7d568aebf88df03179f63c5ff"}
Oct 30 07:04:59.724951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.724867    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn" podStartSLOduration=1.555055038 podStartE2EDuration="1.724849071s" totalImagesPullingTime="169.794033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:59.7241244 +0000 UTC m=+460.156888294" watchObservedRunningTime="2026-10-30 07:04:59.724849071 +0000 UTC m=+460.157612964"
Oct 30 07:04:59.941500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.941412    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"]
Oct 30 07:04:59.944916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.944890    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:04:59.967060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.967029    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"]
Oct 30 07:04:59.968188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.968166    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 07:04:59.985617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:04:59.985594    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:05:00.007117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.007083    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:00.052171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052127    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.052171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052168    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.052408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052190    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94d72-1c5c-414e-b2e5-189467c1edab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.052408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052381    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktvb\" (UniqueName: \"kubernetes.io/projected/2dc94d72-1c5c-414e-b2e5-189467c1edab-kube-api-access-nktvb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.052470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052418    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.052470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.052446    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.153856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.153818    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktvb\" (UniqueName: \"kubernetes.io/projected/2dc94d72-1c5c-414e-b2e5-189467c1edab-kube-api-access-nktvb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.153856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.153862    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.153984    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154086    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154105    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94d72-1c5c-414e-b2e5-189467c1edab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154310    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154406    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.154498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.154475    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.156193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.156167    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc94d72-1c5c-414e-b2e5-189467c1edab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.156688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.156670    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc94d72-1c5c-414e-b2e5-189467c1edab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.178819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.178781    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktvb\" (UniqueName: \"kubernetes.io/projected/2dc94d72-1c5c-414e-b2e5-189467c1edab-kube-api-access-nktvb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d\" (UID: \"2dc94d72-1c5c-414e-b2e5-189467c1edab\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.257189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.257088    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:00.558561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.558498    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"]
Oct 30 07:05:00.560540 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:00.560510    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc94d72_1c5c_414e_b2e5_189467c1edab.slice/crio-5382b25ee648dbe8e4ef8579da6906b3701f3ad755c6d0d1e5010502d2a72f3d WatchSource:0}: Error finding container 5382b25ee648dbe8e4ef8579da6906b3701f3ad755c6d0d1e5010502d2a72f3d: Status 404 returned error can't find the container with id 5382b25ee648dbe8e4ef8579da6906b3701f3ad755c6d0d1e5010502d2a72f3d
Oct 30 07:05:00.685597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:00.685563    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" event={"ID":"2dc94d72-1c5c-414e-b2e5-189467c1edab","Type":"ContainerStarted","Data":"5382b25ee648dbe8e4ef8579da6906b3701f3ad755c6d0d1e5010502d2a72f3d"}
Oct 30 07:05:01.694482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:01.694441    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" event={"ID":"2dc94d72-1c5c-414e-b2e5-189467c1edab","Type":"ContainerStarted","Data":"48df4904c543be6185c7e542c7fd5ded2348d0966ff6683a9d180845b894e879"}
Oct 30 07:05:01.737165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:01.737112    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podStartSLOduration=2.558063646 podStartE2EDuration="2.737100016s" totalImagesPullingTime="179.03637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:01.736759494 +0000 UTC m=+462.169523386" watchObservedRunningTime="2026-10-30 07:05:01.737100016 +0000 UTC m=+462.169863908"
Oct 30 07:05:02.257842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:02.257801    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:03.300884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:03.300847    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:05:03.313348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:03.313324    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:05:03.714698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:03.714673    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-95n5t"
Oct 30 07:05:07.515980 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:05:07.515940    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:05:08.984968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:08.984931    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:05:08.997566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:08.997542    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:05:09.742591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:09.742561    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn"
Oct 30 07:05:10.257509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:10.257465    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:10.270360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:10.270321    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:10.751245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:10.751216    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:05:17.099484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:17.099442    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:17.240717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:17.240675    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:18.758767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:18.758737    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:18.878682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:18.878643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:22.360945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.360887    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:05:22.366527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.366504    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.399719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.399687    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sp9gr\""
Oct 30 07:05:22.400956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.400935    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:05:22.470431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470395    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470459    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470579    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470634    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470687    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470706    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470758    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470798    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.470849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.470822    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8fr\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.571778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571740    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.571778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571781    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571807    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571848    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571880    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8fr\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.571910    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572119    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572202    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572214    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572263    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572322    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572522    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572544    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.572991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.572964    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.574524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.574500    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.574952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.574936    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.595982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.595942    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.596147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.596045    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8fr\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\") pod \"maas-default-gateway-openshift-default-6dcf757899-8srb4\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.679992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.679956    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:22.992541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.992419    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:05:22.995093 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:22.995064    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209e1900_beb5_4db6_b6ef_93f51023684e.slice/crio-b17d90d614a36c72a89ba81a5f8d6e08cf63f4df6bd2ba76d224533c311531ef WatchSource:0}: Error finding container b17d90d614a36c72a89ba81a5f8d6e08cf63f4df6bd2ba76d224533c311531ef: Status 404 returned error can't find the container with id b17d90d614a36c72a89ba81a5f8d6e08cf63f4df6bd2ba76d224533c311531ef
Oct 30 07:05:22.997118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:22.997097    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:05:23.796995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:23.796961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" event={"ID":"209e1900-beb5-4db6-b6ef-93f51023684e","Type":"ContainerStarted","Data":"b17d90d614a36c72a89ba81a5f8d6e08cf63f4df6bd2ba76d224533c311531ef"}
Oct 30 07:05:25.938496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:25.938458    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:25.938959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:25.938547    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:25.938959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:25.938640    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:26.815698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:26.815667    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" event={"ID":"209e1900-beb5-4db6-b6ef-93f51023684e","Type":"ContainerStarted","Data":"5829f826d0fcff4c8d44ac1d40d7216566b961dd3d387ae042f4e48b782651c6"}
Oct 30 07:05:27.692476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:27.692440    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:27.692993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:27.692612    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:27.695304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:27.695270    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:27.794526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:27.794487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:28.687319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:28.680826    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:28.687319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:28.680888    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:29.692482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:29.692440    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:29.692918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:29.692502    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:30.692318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:30.691416    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:30.692318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:30.691483    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:31.683636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:31.683590    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:31.684225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:31.684190    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:32.681002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:32.680963    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 07:05:32.681181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:32.681019    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 07:05:34.185689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:34.185654    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:34.186165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:34.185949    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:34.187029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:34.187000    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:05:34.246738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:34.246683    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podStartSLOduration=9.306421204 podStartE2EDuration="12.246670611s" totalImagesPullingTime="2.940249407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:26.882044009 +0000 UTC m=+487.314807904" watchObservedRunningTime="2026-10-30 07:05:34.246670611 +0000 UTC m=+494.679434503"
Oct 30 07:05:37.345248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.345206    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"]
Oct 30 07:05:37.348918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.348894    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.390965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.390935    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"]
Oct 30 07:05:37.392696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.392667    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:05:37.396190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.396171    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.404711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.404677    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.404843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.404734    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.404843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.404761    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.404843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.404793    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c79cfa-d9a5-4927-a969-42a83e34af9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.404843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.404826    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.406229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.406199    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.406433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.406413    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.406751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.406721    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.408882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.408852    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.408901    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.408929    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409026    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9dz\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-kube-api-access-lz9dz\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.409123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409056    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409079    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2qf\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409105    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409127    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.409283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409171    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.409283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.409200    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.416892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.416861    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:05:37.510011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.509968    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.510477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.510440    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.510627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.510481    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c79cfa-d9a5-4927-a969-42a83e34af9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.510880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.510853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.510941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.510914    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.510964    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511008    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511038    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511065    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511089    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511113    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511146    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9dz\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-kube-api-access-lz9dz\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511177    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511203    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2qf\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511232    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511258    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511322    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511350    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511475    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.511730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511542    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511707    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.511924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.511783    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.512313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512265    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.512313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512285    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.512525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512502    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.512584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512530    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.512584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512545    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c79cfa-d9a5-4927-a969-42a83e34af9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.512584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.512557    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c79cfa-d9a5-4927-a969-42a83e34af9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.514251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.514228    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.514501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.514482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.516150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.516129    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.519286 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:05:37.519260    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:05:37.536066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.536022    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.536066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.536059    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.536237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.536107    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2qf\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\") pod \"maas-default-gateway-openshift-default-6dcf757899-kwlpn\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.536237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.536151    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9dz\" (UniqueName: \"kubernetes.io/projected/22c79cfa-d9a5-4927-a969-42a83e34af9d-kube-api-access-lz9dz\") pod \"maas-default-gateway-openshift-default-6dcf757899-hrjtp\" (UID: \"22c79cfa-d9a5-4927-a969-42a83e34af9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.662686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.662642    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:37.716215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.716106    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:37.979835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.979806    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"]
Oct 30 07:05:37.981196 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:37.981162    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c79cfa_d9a5_4927_a969_42a83e34af9d.slice/crio-24269112c9efacc60c42739d3ada099593e4bcbf4eba40cf88d701790abaaee0 WatchSource:0}: Error finding container 24269112c9efacc60c42739d3ada099593e4bcbf4eba40cf88d701790abaaee0: Status 404 returned error can't find the container with id 24269112c9efacc60c42739d3ada099593e4bcbf4eba40cf88d701790abaaee0
Oct 30 07:05:37.984442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.984407    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:37.984522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.984474    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:37.984522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:37.984505    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:38.026696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.026664    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:05:38.029150 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:38.029088    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52155c4_081e_4709_b6a6_6b31eeb6c7d7.slice/crio-5adf60695d3f6ecdfd1cb7fd489b4aaa5664c25baccf30d45c0e3ebe3cd07fb4 WatchSource:0}: Error finding container 5adf60695d3f6ecdfd1cb7fd489b4aaa5664c25baccf30d45c0e3ebe3cd07fb4: Status 404 returned error can't find the container with id 5adf60695d3f6ecdfd1cb7fd489b4aaa5664c25baccf30d45c0e3ebe3cd07fb4
Oct 30 07:05:38.032817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.032775    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:38.032909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.032865    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:38.032909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.032896    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:38.902317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.901506    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" event={"ID":"22c79cfa-d9a5-4927-a969-42a83e34af9d","Type":"ContainerStarted","Data":"f7bb20f007eef0a128fce89a8ed8be60f4debf5541eb8ce111ea3c4d39d8d084"}
Oct 30 07:05:38.902317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.901595    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" event={"ID":"22c79cfa-d9a5-4927-a969-42a83e34af9d","Type":"ContainerStarted","Data":"24269112c9efacc60c42739d3ada099593e4bcbf4eba40cf88d701790abaaee0"}
Oct 30 07:05:38.920901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.920867    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" event={"ID":"b52155c4-081e-4709-b6a6-6b31eeb6c7d7","Type":"ContainerStarted","Data":"726d974006f7ca47d44c129e9d62979d0c098975be8d7bf3d9107b859bc804a3"}
Oct 30 07:05:38.921080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.920924    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" event={"ID":"b52155c4-081e-4709-b6a6-6b31eeb6c7d7","Type":"ContainerStarted","Data":"5adf60695d3f6ecdfd1cb7fd489b4aaa5664c25baccf30d45c0e3ebe3cd07fb4"}
Oct 30 07:05:38.964717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:38.964668    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podStartSLOduration=1.964654918 podStartE2EDuration="1.964654918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:38.9643799 +0000 UTC m=+499.397143793" watchObservedRunningTime="2026-10-30 07:05:38.964654918 +0000 UTC m=+499.397418805"
Oct 30 07:05:39.662962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.662927    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:39.664461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.664433    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:39.664585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.664512    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:39.718321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.717663    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:39.722312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.719482    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:39.722312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:39.719565    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:40.663279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:40.663241    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:40.663789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:40.663328    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:40.717877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:40.717834    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:40.718089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:40.717903    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:41.663885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:41.663848    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:41.664319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:41.663907    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:41.718988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:41.718954    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:41.719245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:41.719219    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:42.663186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:42.663145    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:42.663391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:42.663213    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:42.717912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:42.717752    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:42.717912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:42.717823    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:43.663549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:43.663512    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:43.663741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:43.663565    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:43.719316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:43.718742    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:43.719316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:43.718804    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:44.663484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:44.663438    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hrjtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 07:05:44.663672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:44.663509    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp" podUID="22c79cfa-d9a5-4927-a969-42a83e34af9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 07:05:44.716998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:44.716954    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 07:05:44.717187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:44.717028    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 07:05:46.352462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.352423    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:46.352995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.352953    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:46.354168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.354149    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hrjtp"
Oct 30 07:05:46.412136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.412072    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podStartSLOduration=9.412053524 podStartE2EDuration="9.412053524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:39.018239681 +0000 UTC m=+499.451003575" watchObservedRunningTime="2026-10-30 07:05:46.412053524 +0000 UTC m=+506.844817418"
Oct 30 07:05:46.457766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.457726    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:46.458985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.458823    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:46.459596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:46.459576    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:05:48.022543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.022494    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:48.140773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.140731    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:48.287878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.287787    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:48.407849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.407813    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:48.610220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.610123    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:48.973117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:48.973076    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:49.088128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:49.088097    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:49.375045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:49.375010    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:49.500659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:49.500627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:50.105457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:50.105419    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:50.596271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:50.596239    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:50.782568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:50.782531    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.122461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.122424    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.254054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.254016    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.434997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.434915    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.557690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.557653    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.678682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.678642    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:52.798409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:52.798282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:53.610449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.610415    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:53.747726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.747676    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:53.785578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.785542    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:05:53.789504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.789484    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:53.807526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.807496    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:05:53.818224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.818191    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-414c5686\""
Oct 30 07:05:53.880832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.880796    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:05:53.885083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.885055    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:53.891148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.890691    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:53.891148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.890858    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmd52\" (UniqueName: \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:53.906603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.904623    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-414c5686-dockercfg-6jqnh\""
Oct 30 07:05:53.915908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.915872    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:05:53.991642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.991606    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrv8\" (UniqueName: \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:53.991824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.991673    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmd52\" (UniqueName: \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:53.991824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.991745    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:53.991824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.991803    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:53.992721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:53.992695    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:54.015611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.015577    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmd52\" (UniqueName: \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\") pod \"payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:54.092713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.092676    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:54.092919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.092824    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrv8\" (UniqueName: \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:54.093540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.093518    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:54.103516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.103490    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:54.117113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.117088    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrv8\" (UniqueName: \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\") pod \"payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") " pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:54.144925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.144425    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:54.220443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.220412    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:54.289923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.289887    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:54.402677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.402644    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:05:54.404260 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:54.404232    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c23ffe_4aa3_48f7_aae5_f146d0406304.slice/crio-d5aa146fcdf38060c34fb514ce7a1d58a4a238c195b3aab269527fe1fc4a257b WatchSource:0}: Error finding container d5aa146fcdf38060c34fb514ce7a1d58a4a238c195b3aab269527fe1fc4a257b: Status 404 returned error can't find the container with id d5aa146fcdf38060c34fb514ce7a1d58a4a238c195b3aab269527fe1fc4a257b
Oct 30 07:05:54.537249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:54.537180    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:05:54.539589 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:54.539560    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e06f1f_d7b4_4fc6_9aa0_d7e6a99a2494.slice/crio-aa1c510f284e511fcb65504ddb75fc133a52b8f18cb81252dd174d8a9ebdd609 WatchSource:0}: Error finding container aa1c510f284e511fcb65504ddb75fc133a52b8f18cb81252dd174d8a9ebdd609: Status 404 returned error can't find the container with id aa1c510f284e511fcb65504ddb75fc133a52b8f18cb81252dd174d8a9ebdd609
Oct 30 07:05:55.001742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.001691    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" event={"ID":"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494","Type":"ContainerStarted","Data":"aa1c510f284e511fcb65504ddb75fc133a52b8f18cb81252dd174d8a9ebdd609"}
Oct 30 07:05:55.004176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.004143    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" event={"ID":"25c23ffe-4aa3-48f7-aae5-f146d0406304","Type":"ContainerStarted","Data":"d5aa146fcdf38060c34fb514ce7a1d58a4a238c195b3aab269527fe1fc4a257b"}
Oct 30 07:05:55.248044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.248002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:55.351027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.350940    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:05:55.355098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.355071    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.372530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.372498    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:05:55.377570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.377532    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:55.383441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.383071    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a9eddc5d\""
Oct 30 07:05:55.383441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.383169    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a9eddc5d-dockercfg-bp5nh\""
Oct 30 07:05:55.398270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.398229    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:05:55.402913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.402882    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.406634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.406417    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnwt\" (UniqueName: \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.407059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.406947    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.422854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.422820    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:05:55.511520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.508023    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnwt\" (UniqueName: \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.516041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.508228    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.516041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.514468    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9s5\" (UniqueName: \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.516041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.514519    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.516041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.515553    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.535123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.535054    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnwt\" (UniqueName: \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\") pod \"payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") " pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.585633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.585591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:55.615703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.615613    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.615703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.615658    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9s5\" (UniqueName: \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.616593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.616568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.651437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.651401    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9s5\" (UniqueName: \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\") pod \"payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:55.674207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.674174    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:55.706323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.706268    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:55.732036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:55.732001    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:56.012488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.012416    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:05:56.015115 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:56.015076    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06657cd5_fe4a_4b28_b8cf_e8a545adaa61.slice/crio-895d8c3186776ba34b18432a97d558aedc7f72a51c8bb8acbcca474e3bc07e5f WatchSource:0}: Error finding container 895d8c3186776ba34b18432a97d558aedc7f72a51c8bb8acbcca474e3bc07e5f: Status 404 returned error can't find the container with id 895d8c3186776ba34b18432a97d558aedc7f72a51c8bb8acbcca474e3bc07e5f
Oct 30 07:05:56.072087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.072052    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:05:56.075338 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:56.075205    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20eeb1a_0d19_44f7_a08f_edcea8bdb8b7.slice/crio-a5511424a5941ce29cf7e82f0e26099daf533e5ba3b906048852cfac8053043f WatchSource:0}: Error finding container a5511424a5941ce29cf7e82f0e26099daf533e5ba3b906048852cfac8053043f: Status 404 returned error can't find the container with id a5511424a5941ce29cf7e82f0e26099daf533e5ba3b906048852cfac8053043f
Oct 30 07:05:56.133211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.133174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.261711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.261625    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.307750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.307717    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:05:56.325452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.325418    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.334448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.334176    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:05:56.343387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.341545    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a1a08662-dockercfg-x9zst\""
Oct 30 07:05:56.343387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.341634    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a1a08662\""
Oct 30 07:05:56.375190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.374094    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.399263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.399224    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:05:56.407064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.406688    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.427411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.426222    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvt7\" (UniqueName: \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.427411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.426340    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.442645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.438399    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:05:56.527744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.527661    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.527744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.527724    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlrf\" (UniqueName: \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.527974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.527795    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.527974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.527930    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvt7\" (UniqueName: \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.529210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.529184    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.558479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.558442    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvt7\" (UniqueName: \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\") pod \"payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.628815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.628760    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlrf\" (UniqueName: \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.629008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.628896    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.630528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.630501    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.646349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.646282    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:56.655733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.655699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.655958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.655853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlrf\" (UniqueName: \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\") pod \"payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") " pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.741211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.741183    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:56.776822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.776774    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.983098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.983054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:56.999756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:56.999723    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:05:57.001383 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:57.001353    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aba1948_31a6_4fa0_b40f_65d813fe5e1c.slice/crio-e2341c5537a2cb2d2947654477745ebb4e2e9a3b3193202b930e7d923a615c52 WatchSource:0}: Error finding container e2341c5537a2cb2d2947654477745ebb4e2e9a3b3193202b930e7d923a615c52: Status 404 returned error can't find the container with id e2341c5537a2cb2d2947654477745ebb4e2e9a3b3193202b930e7d923a615c52
Oct 30 07:05:57.023671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.023599    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" event={"ID":"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7","Type":"ContainerStarted","Data":"a5511424a5941ce29cf7e82f0e26099daf533e5ba3b906048852cfac8053043f"}
Oct 30 07:05:57.026703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.026669    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" event={"ID":"06657cd5-fe4a-4b28-b8cf-e8a545adaa61","Type":"ContainerStarted","Data":"895d8c3186776ba34b18432a97d558aedc7f72a51c8bb8acbcca474e3bc07e5f"}
Oct 30 07:05:57.032030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.031989    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" event={"ID":"8aba1948-31a6-4fa0-b40f-65d813fe5e1c","Type":"ContainerStarted","Data":"e2341c5537a2cb2d2947654477745ebb4e2e9a3b3193202b930e7d923a615c52"}
Oct 30 07:05:57.091204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.091167    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:05:57.510740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.509539    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:05:57.514658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.514628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.534539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.534508    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-415d1917-openshift-default-dockercfg-2hbwq\""
Oct 30 07:05:57.562887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.562851    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:05:57.570229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.570199    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:57.661379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661344    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661406    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661483    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtkf\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661578    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661617    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661732    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661793    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661867    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.661946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.661909    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.755281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.697278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:57.763597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.763516    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.763597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.763572    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.763597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.763598    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtkf\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.763843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.763783    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.763943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.763918    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764020    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764093    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764111    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764132    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764172    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764629    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764796    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.764985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.764899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.765047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.765010    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.766954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.766906    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.767546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.767523    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.788962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.788906    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.792209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.792177    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtkf\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\") pod \"e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") " pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:57.864987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:57.864962    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:58.040258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.040162    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" event={"ID":"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494","Type":"ContainerStarted","Data":"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"}
Oct 30 07:05:58.040690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.040335    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:05:58.041831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.041800    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" event={"ID":"26d5bd40-a0c1-4b54-8f4c-105ab10fff54","Type":"ContainerStarted","Data":"badb5e6b4e1e75e99925ac923cc887cc6dad497d63f9e563ac453821fd795567"}
Oct 30 07:05:58.041955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.041849    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" event={"ID":"26d5bd40-a0c1-4b54-8f4c-105ab10fff54","Type":"ContainerStarted","Data":"ec6c8a7b9c63c6cf10b71c5ded7618dfccef67ffc2e45543578eee0ce249ce6b"}
Oct 30 07:05:58.041955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.041882    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:05:58.043443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.043410    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" event={"ID":"25c23ffe-4aa3-48f7-aae5-f146d0406304","Type":"ContainerStarted","Data":"caceb77fcc0b930fb5b2236018c638cb1c75fa2fe2a2c8588ec8dc09c3fe177b"}
Oct 30 07:05:58.043558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.043477    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:05:58.045066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.045040    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" event={"ID":"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7","Type":"ContainerStarted","Data":"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"}
Oct 30 07:05:58.045211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.045091    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:05:58.046664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.046637    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" event={"ID":"06657cd5-fe4a-4b28-b8cf-e8a545adaa61","Type":"ContainerStarted","Data":"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"}
Oct 30 07:05:58.046765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.046685    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:05:58.048140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.048118    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" event={"ID":"8aba1948-31a6-4fa0-b40f-65d813fe5e1c","Type":"ContainerStarted","Data":"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"}
Oct 30 07:05:58.048261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.048248    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:05:58.082440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.082375    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" podStartSLOduration=1.812170596 podStartE2EDuration="5.082357138s" totalImagesPullingTime="3.270186542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.081492316 +0000 UTC m=+518.514256210" watchObservedRunningTime="2026-10-30 07:05:58.082357138 +0000 UTC m=+518.515121033"
Oct 30 07:05:58.123102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.122936    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" podStartSLOduration=1.293748062 podStartE2EDuration="3.122915949s" totalImagesPullingTime="1.829167887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.12221812 +0000 UTC m=+518.554982012" watchObservedRunningTime="2026-10-30 07:05:58.122915949 +0000 UTC m=+518.555679840"
Oct 30 07:05:58.163653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.163575    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" podStartSLOduration=1.7436162899999998 podStartE2EDuration="2.163558454s" totalImagesPullingTime="419.942164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.162916492 +0000 UTC m=+518.595680385" watchObservedRunningTime="2026-10-30 07:05:58.163558454 +0000 UTC m=+518.596322349"
Oct 30 07:05:58.187102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.187065    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:05:58.189446 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:58.189421    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8746406c_0cb4_4a13_9cd8_a278b476dc12.slice/crio-894a0ed21c495f856f12aa405bc09e5ecd917d094d5728f65ee7607541f5c92c WatchSource:0}: Error finding container 894a0ed21c495f856f12aa405bc09e5ecd917d094d5728f65ee7607541f5c92c: Status 404 returned error can't find the container with id 894a0ed21c495f856f12aa405bc09e5ecd917d094d5728f65ee7607541f5c92c
Oct 30 07:05:58.191996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.191951    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:58.192079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.192055    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:58.192123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.192105    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:05:58.249490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.249429    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" podStartSLOduration=1.419828944 podStartE2EDuration="3.249407758s" totalImagesPullingTime="1.829578814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.247589202 +0000 UTC m=+518.680353098" watchObservedRunningTime="2026-10-30 07:05:58.249407758 +0000 UTC m=+518.682171653"
Oct 30 07:05:58.249840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.249804    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" podStartSLOduration=2.137360886 podStartE2EDuration="5.249792204s" totalImagesPullingTime="3.112431318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.203682989 +0000 UTC m=+518.636446882" watchObservedRunningTime="2026-10-30 07:05:58.249792204 +0000 UTC m=+518.682556096"
Oct 30 07:05:58.290232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.290181    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" podStartSLOduration=1.668648041 podStartE2EDuration="2.29016571s" totalImagesPullingTime="621.517669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:58.288668318 +0000 UTC m=+518.721432210" watchObservedRunningTime="2026-10-30 07:05:58.29016571 +0000 UTC m=+518.722929602"
Oct 30 07:05:58.515970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.515931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:58.640946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.640911    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:58.755916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.755874    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:05:58.872900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.872868    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:05:58.877470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.877443    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:58.895391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.895356    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b764790-dockercfg-x6ljq\""
Oct 30 07:05:58.898588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.898555    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:05:58.931176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.931147    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b764790\""
Oct 30 07:05:58.977660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.977623    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvkl\" (UniqueName: \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:58.977847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:58.977671    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.055104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.055048    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerStarted","Data":"a0acdd37652a9ac2697d52d28ae7a1e0241bc9edc99d5bed40846385c425f6b4"}
Oct 30 07:05:59.055600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.055136    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerStarted","Data":"894a0ed21c495f856f12aa405bc09e5ecd917d094d5728f65ee7607541f5c92c"}
Oct 30 07:05:59.078988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.078945    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvkl\" (UniqueName: \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.079175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.079001    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.079785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.079764    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.102861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.102804    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podStartSLOduration=2.102789869 podStartE2EDuration="2.102789869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:59.101155922 +0000 UTC m=+519.533919817" watchObservedRunningTime="2026-10-30 07:05:59.102789869 +0000 UTC m=+519.535553762"
Oct 30 07:05:59.106832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.106807    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvkl\" (UniqueName: \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\") pod \"payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") " pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.220494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.220037    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:05:59.535937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.535905    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:05:59.536957 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:05:59.536925    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48d84e_e105_4e63_b587_33769433075b.slice/crio-cb1de29f4c2ee55ca4c237259cdbd612b920c635d048a259013a1a4a1013d449 WatchSource:0}: Error finding container cb1de29f4c2ee55ca4c237259cdbd612b920c635d048a259013a1a4a1013d449: Status 404 returned error can't find the container with id cb1de29f4c2ee55ca4c237259cdbd612b920c635d048a259013a1a4a1013d449
Oct 30 07:05:59.865451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.865371    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:05:59.870555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:05:59.870528    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:00.062021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.061973    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" event={"ID":"dc48d84e-e105-4e63-b587-33769433075b","Type":"ContainerStarted","Data":"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"}
Oct 30 07:06:00.062482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.062034    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" event={"ID":"dc48d84e-e105-4e63-b587-33769433075b","Type":"ContainerStarted","Data":"cb1de29f4c2ee55ca4c237259cdbd612b920c635d048a259013a1a4a1013d449"}
Oct 30 07:06:00.062482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.062241    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:00.063279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.063261    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:00.104715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.104654    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" podStartSLOduration=2.104640895 podStartE2EDuration="2.104640895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:00.104275363 +0000 UTC m=+520.537039257" watchObservedRunningTime="2026-10-30 07:06:00.104640895 +0000 UTC m=+520.537404788"
Oct 30 07:06:00.631540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:00.631503    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:01.067866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:01.067772    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:06:01.572193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:01.572149    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:02.613546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.613501    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:02.730625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.730568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:02.782468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.782434    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:02.786021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.785995    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.805888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.805831    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:02.816034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.816006    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-415d1917\""
Oct 30 07:06:02.816034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.816008    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-415d1917-dockercfg-lqsjk\""
Oct 30 07:06:02.818646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.818621    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.818749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.818679    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsvv\" (UniqueName: \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.844529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.844491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:02.920353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.920314    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.920537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.920427    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsvv\" (UniqueName: \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.921760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.921728    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.950170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.950133    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsvv\" (UniqueName: \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\") pod \"payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:02.955225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:02.955186    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:03.099454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:03.099407    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:03.403775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:03.403748    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:03.404893 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:03.404865    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893d3ca1_e493_434d_b95d_b0ce544086a8.slice/crio-ce2d3f79b239fe74dc9871ef3bbe1d7a78f74dd9a02fecd30a30e05d483c0a2e WatchSource:0}: Error finding container ce2d3f79b239fe74dc9871ef3bbe1d7a78f74dd9a02fecd30a30e05d483c0a2e: Status 404 returned error can't find the container with id ce2d3f79b239fe74dc9871ef3bbe1d7a78f74dd9a02fecd30a30e05d483c0a2e
Oct 30 07:06:04.084365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:04.084330    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" event={"ID":"893d3ca1-e493-434d-b95d-b0ce544086a8","Type":"ContainerStarted","Data":"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"}
Oct 30 07:06:04.084817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:04.084390    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" event={"ID":"893d3ca1-e493-434d-b95d-b0ce544086a8","Type":"ContainerStarted","Data":"ce2d3f79b239fe74dc9871ef3bbe1d7a78f74dd9a02fecd30a30e05d483c0a2e"}
Oct 30 07:06:04.084817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:04.084486    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:04.126829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:04.126775    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" podStartSLOduration=2.126760422 podStartE2EDuration="2.126760422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:04.12515199 +0000 UTC m=+524.557915894" watchObservedRunningTime="2026-10-30 07:06:04.126760422 +0000 UTC m=+524.559524314"
Oct 30 07:06:04.748868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:04.748830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:06.682488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:06.682452    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:06.806107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:06.806070    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:06.968153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:06.968066    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:07.373586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.373550    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"]
Oct 30 07:06:07.377420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.377401    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.392796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.392764    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:06:07.395434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.395405    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"]
Oct 30 07:06:07.405671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.405644    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:06:07.446494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.446457    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:06:07.446675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.446539    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:06:07.471107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.471066    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.471107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.471106    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfb5\" (UniqueName: \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.471381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.471240    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.529833 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:07.529695    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:06:07.572195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.572159    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.572400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.572274    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.572400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.572314    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfb5\" (UniqueName: \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.572975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.572948    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.574502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.574481    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.600842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.600806    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfb5\" (UniqueName: \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\") pod \"maas-api-revoke-keys-e2e-migrate-414c5686-kprp5\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.691038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.690960    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:07.997206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:07.997174    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"]
Oct 30 07:06:07.998021 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:07.997992    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9187a065_b98c_4813_a747_83a3211fb92d.slice/crio-cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f WatchSource:0}: Error finding container cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f: Status 404 returned error can't find the container with id cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f
Oct 30 07:06:08.104911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:08.104877    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5" event={"ID":"9187a065-b98c-4813-a747-83a3211fb92d","Type":"ContainerStarted","Data":"cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f"}
Oct 30 07:06:08.772228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:08.772194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:09.059078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.058927    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:06:09.059235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.059119    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:06:09.059235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.059161    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:06:09.059235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.059197    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:06:09.059474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.059458    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:06:09.059568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:09.059495    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:06:10.727950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:10.727913    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:10.870836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:10.865817    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:10.984648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:10.984572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:12.074585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:12.074559    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:06:12.130335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:12.130283    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5" event={"ID":"9187a065-b98c-4813-a747-83a3211fb92d","Type":"ContainerStarted","Data":"9f0700353067860a6f08077278995f5c84c3be1e6f94f24cc05578ea2546940d"}
Oct 30 07:06:12.173840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:12.173776    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5" podStartSLOduration=1.743129847 podStartE2EDuration="5.173755412s" totalImagesPullingTime="3.430625565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:12.17170757 +0000 UTC m=+532.604471463" watchObservedRunningTime="2026-10-30 07:06:12.173755412 +0000 UTC m=+532.606519306"
Oct 30 07:06:12.843974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:12.843932    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:13.137378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.137341    6738 generic.go:356] "Generic (PLEG): container finished" podID="9187a065-b98c-4813-a747-83a3211fb92d" containerID="9f0700353067860a6f08077278995f5c84c3be1e6f94f24cc05578ea2546940d" exitCode=0
Oct 30 07:06:13.137816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.137407    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5" event={"ID":"9187a065-b98c-4813-a747-83a3211fb92d","Type":"ContainerDied","Data":"9f0700353067860a6f08077278995f5c84c3be1e6f94f24cc05578ea2546940d"}
Oct 30 07:06:13.196211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.196173    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"]
Oct 30 07:06:13.200929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.200904    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.220142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.220107    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"]
Oct 30 07:06:13.243497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.243454    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.243680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.243571    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.243680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.243654    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kc9h\" (UniqueName: \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.345219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.345181    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.345439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.345247    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9h\" (UniqueName: \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.345439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.345358    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.346004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.345985    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.347501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.347482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.369213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.369185    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc9h\" (UniqueName: \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\") pod \"maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.514037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.513944    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:13.814111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:13.814086    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"]
Oct 30 07:06:13.815346 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:13.815313    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21f1bff_98a0_4c64_89dd_b10ea95dcad4.slice/crio-5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019 WatchSource:0}: Error finding container 5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019: Status 404 returned error can't find the container with id 5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019
Oct 30 07:06:14.143660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.143619    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq" event={"ID":"c21f1bff-98a0-4c64-89dd-b10ea95dcad4","Type":"ContainerStarted","Data":"46353d5fff893ee79f832d6ed459af639410957108ab8d46ec3e54dbe1c4df4c"}
Oct 30 07:06:14.144153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.143670    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq" event={"ID":"c21f1bff-98a0-4c64-89dd-b10ea95dcad4","Type":"ContainerStarted","Data":"5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019"}
Oct 30 07:06:14.185742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.185680    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq" podStartSLOduration=1.185664299 podStartE2EDuration="1.185664299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:14.184546956 +0000 UTC m=+534.617310851" watchObservedRunningTime="2026-10-30 07:06:14.185664299 +0000 UTC m=+534.618428192"
Oct 30 07:06:14.256571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.256544    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:14.356973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.356942    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\" (UniqueName: \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\") pod \"9187a065-b98c-4813-a747-83a3211fb92d\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") "
Oct 30 07:06:14.357110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.357093    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\") pod \"9187a065-b98c-4813-a747-83a3211fb92d\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") "
Oct 30 07:06:14.357172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.357148    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\") pod \"9187a065-b98c-4813-a747-83a3211fb92d\" (UID: \"9187a065-b98c-4813-a747-83a3211fb92d\") "
Oct 30 07:06:14.357481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.357457    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca" pod "9187a065-b98c-4813-a747-83a3211fb92d" (UID: "9187a065-b98c-4813-a747-83a3211fb92d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:14.357611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.357595    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:14.359328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.359289    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca" pod "9187a065-b98c-4813-a747-83a3211fb92d" (UID: "9187a065-b98c-4813-a747-83a3211fb92d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:14.359414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.359397    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5" pod "9187a065-b98c-4813-a747-83a3211fb92d" (UID: "9187a065-b98c-4813-a747-83a3211fb92d"). InnerVolumeSpecName "kube-api-access-2lfb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:14.459214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.459171    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfb5\" (UniqueName: \"kubernetes.io/projected/9187a065-b98c-4813-a747-83a3211fb92d-kube-api-access-2lfb5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:14.459214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.459216    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9187a065-b98c-4813-a747-83a3211fb92d-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:14.759445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:14.759409    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:15.093196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.093106    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:15.152417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.152325    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5" event={"ID":"9187a065-b98c-4813-a747-83a3211fb92d","Type":"ContainerDied","Data":"cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f"}
Oct 30 07:06:15.152417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.152387    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8d9ec1fff4acf4b52e4daa4de316583f86bf3c22f28d8b186f5f77a95a659f"
Oct 30 07:06:15.152943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.152622    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"
Oct 30 07:06:15.158517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.158484    6738 generic.go:356] "Generic (PLEG): container finished" podID="c21f1bff-98a0-4c64-89dd-b10ea95dcad4" containerID="46353d5fff893ee79f832d6ed459af639410957108ab8d46ec3e54dbe1c4df4c" exitCode=0
Oct 30 07:06:15.158683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:15.158566    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq" event={"ID":"c21f1bff-98a0-4c64-89dd-b10ea95dcad4","Type":"ContainerDied","Data":"46353d5fff893ee79f832d6ed459af639410957108ab8d46ec3e54dbe1c4df4c"}
Oct 30 07:06:16.301481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.301450    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:16.379529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.379492    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\") pod \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") "
Oct 30 07:06:16.379529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.379541    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\") pod \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") "
Oct 30 07:06:16.379777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.379577    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\" (UniqueName: \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\") pod \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\" (UID: \"c21f1bff-98a0-4c64-89dd-b10ea95dcad4\") "
Oct 30 07:06:16.380042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.380015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca" pod "c21f1bff-98a0-4c64-89dd-b10ea95dcad4" (UID: "c21f1bff-98a0-4c64-89dd-b10ea95dcad4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:16.381829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.381803    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca" pod "c21f1bff-98a0-4c64-89dd-b10ea95dcad4" (UID: "c21f1bff-98a0-4c64-89dd-b10ea95dcad4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:16.381968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.381860    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h" pod "c21f1bff-98a0-4c64-89dd-b10ea95dcad4" (UID: "c21f1bff-98a0-4c64-89dd-b10ea95dcad4"). InnerVolumeSpecName "kube-api-access-8kc9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:16.482126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.482034    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:16.482126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.482083    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:16.482126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.482099    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kc9h\" (UniqueName: \"kubernetes.io/projected/c21f1bff-98a0-4c64-89dd-b10ea95dcad4-kube-api-access-8kc9h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:16.842861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:16.842778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:17.171491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:17.171458    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"
Oct 30 07:06:17.171672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:17.171497    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq" event={"ID":"c21f1bff-98a0-4c64-89dd-b10ea95dcad4","Type":"ContainerDied","Data":"5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019"}
Oct 30 07:06:17.171672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:17.171583    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b24aee7dcdf19d154b53e476627100b5c572257c7b2011b34afe65ef80ca019"
Oct 30 07:06:17.815446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:17.815411    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:06:17.862438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:17.862400    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:06:18.066623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.066459    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:18.206314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.205637    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:18.766020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.765982    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"]
Oct 30 07:06:18.766550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766532    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9187a065-b98c-4813-a747-83a3211fb92d" containerName="revoke-keys"
Oct 30 07:06:18.766617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766552    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9187a065-b98c-4813-a747-83a3211fb92d" containerName="revoke-keys"
Oct 30 07:06:18.766617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766563    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c21f1bff-98a0-4c64-89dd-b10ea95dcad4" containerName="revoke-keys"
Oct 30 07:06:18.766617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766569    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21f1bff-98a0-4c64-89dd-b10ea95dcad4" containerName="revoke-keys"
Oct 30 07:06:18.766719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766633    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9187a065-b98c-4813-a747-83a3211fb92d" containerName="revoke-keys"
Oct 30 07:06:18.766719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.766641    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21f1bff-98a0-4c64-89dd-b10ea95dcad4" containerName="revoke-keys"
Oct 30 07:06:18.781536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.781502    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:18.795396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.793705    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"]
Oct 30 07:06:18.803775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.803749    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:06:18.803917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.803762    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:06:18.803977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.803821    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:06:18.811916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.811892    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:06:18.920734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.920701    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:18.921144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.920750    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcgl\" (UniqueName: \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:18.921144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.920878    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:18.930555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:18.930526    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:19.022455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.022363    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcgl\" (UniqueName: \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.022622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.022507    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.022622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.022592    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.023231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.023204    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.024974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.024948    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.046821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.046784    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcgl\" (UniqueName: \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\") pod \"maas-api-revoke-keys-e2e-ait-a1a08662-qth4x\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.110688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.110651    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:19.427277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:19.427249    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"]
Oct 30 07:06:19.428676 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:19.428647    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3241e905_8e8c_473d_b117_bc0754c0638b.slice/crio-ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91 WatchSource:0}: Error finding container ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91: Status 404 returned error can't find the container with id ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91
Oct 30 07:06:20.223363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:20.223319    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x" event={"ID":"3241e905-8e8c-473d-b117-bc0754c0638b","Type":"ContainerStarted","Data":"e48262df1251bdab613c235b13de6c62163ac407878e72a9034cac458c2bb41d"}
Oct 30 07:06:20.223888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:20.223420    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x" event={"ID":"3241e905-8e8c-473d-b117-bc0754c0638b","Type":"ContainerStarted","Data":"ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91"}
Oct 30 07:06:20.267068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:20.267015    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x" podStartSLOduration=2.267002064 podStartE2EDuration="2.267002064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:20.266644424 +0000 UTC m=+540.699408321" watchObservedRunningTime="2026-10-30 07:06:20.267002064 +0000 UTC m=+540.699765956"
Oct 30 07:06:20.955692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:20.955638    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:21.230851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:21.230750    6738 generic.go:356] "Generic (PLEG): container finished" podID="3241e905-8e8c-473d-b117-bc0754c0638b" containerID="e48262df1251bdab613c235b13de6c62163ac407878e72a9034cac458c2bb41d" exitCode=0
Oct 30 07:06:21.230851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:21.230819    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x" event={"ID":"3241e905-8e8c-473d-b117-bc0754c0638b","Type":"ContainerDied","Data":"e48262df1251bdab613c235b13de6c62163ac407878e72a9034cac458c2bb41d"}
Oct 30 07:06:22.409034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.409004    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:22.561043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.560942    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\") pod \"3241e905-8e8c-473d-b117-bc0754c0638b\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") "
Oct 30 07:06:22.561043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.560996    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\") pod \"3241e905-8e8c-473d-b117-bc0754c0638b\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") "
Oct 30 07:06:22.561277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.561053    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\" (UniqueName: \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\") pod \"3241e905-8e8c-473d-b117-bc0754c0638b\" (UID: \"3241e905-8e8c-473d-b117-bc0754c0638b\") "
Oct 30 07:06:22.561444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.561416    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca" pod "3241e905-8e8c-473d-b117-bc0754c0638b" (UID: "3241e905-8e8c-473d-b117-bc0754c0638b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:22.561587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.561569    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:22.563141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.563118    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca" pod "3241e905-8e8c-473d-b117-bc0754c0638b" (UID: "3241e905-8e8c-473d-b117-bc0754c0638b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:22.563338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.563317    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl" pod "3241e905-8e8c-473d-b117-bc0754c0638b" (UID: "3241e905-8e8c-473d-b117-bc0754c0638b"). InnerVolumeSpecName "kube-api-access-cpcgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:22.663045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.662992    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241e905-8e8c-473d-b117-bc0754c0638b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:22.663045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.663033    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpcgl\" (UniqueName: \"kubernetes.io/projected/3241e905-8e8c-473d-b117-bc0754c0638b-kube-api-access-cpcgl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:22.816790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.816672    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" podUID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" containerName="payload-processing" containerID="cri-o://e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb" gracePeriod=25
Oct 30 07:06:22.863400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:22.863346    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" podUID="25c23ffe-4aa3-48f7-aae5-f146d0406304" containerName="payload-pre-processing" containerID="cri-o://caceb77fcc0b930fb5b2236018c638cb1c75fa2fe2a2c8588ec8dc09c3fe177b" gracePeriod=25
Oct 30 07:06:23.152943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.152912    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:06:23.216818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.216773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:23.306834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.306795    6738 generic.go:356] "Generic (PLEG): container finished" podID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" containerID="e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb" exitCode=0
Oct 30 07:06:23.307110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.306896    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"
Oct 30 07:06:23.307409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.306907    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" event={"ID":"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494","Type":"ContainerDied","Data":"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"}
Oct 30 07:06:23.307524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.307488    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt" event={"ID":"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494","Type":"ContainerDied","Data":"aa1c510f284e511fcb65504ddb75fc133a52b8f18cb81252dd174d8a9ebdd609"}
Oct 30 07:06:23.307582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.307543    6738 scope.go:122] "RemoveContainer" containerID="e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"
Oct 30 07:06:23.310212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.310187    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x" event={"ID":"3241e905-8e8c-473d-b117-bc0754c0638b","Type":"ContainerDied","Data":"ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91"}
Oct 30 07:06:23.310265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.310239    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac23d5a00d9f465748e11efc6b2531740739991dc633d16899f9b816f757fd91"
Oct 30 07:06:23.310374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.310361    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"
Oct 30 07:06:23.311422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.311341    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:06:23.311993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.311975    6738 generic.go:356] "Generic (PLEG): container finished" podID="25c23ffe-4aa3-48f7-aae5-f146d0406304" containerID="caceb77fcc0b930fb5b2236018c638cb1c75fa2fe2a2c8588ec8dc09c3fe177b" exitCode=0
Oct 30 07:06:23.312087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.312068    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" event={"ID":"25c23ffe-4aa3-48f7-aae5-f146d0406304","Type":"ContainerDied","Data":"caceb77fcc0b930fb5b2236018c638cb1c75fa2fe2a2c8588ec8dc09c3fe177b"}
Oct 30 07:06:23.317394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.317368    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\") pod \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") "
Oct 30 07:06:23.317529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.317432    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\" (UniqueName: \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\") pod \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\" (UID: \"82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494\") "
Oct 30 07:06:23.317596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.317566    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\") pod \"25c23ffe-4aa3-48f7-aae5-f146d0406304\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") "
Oct 30 07:06:23.317651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.317615    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\" (UniqueName: \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\") pod \"25c23ffe-4aa3-48f7-aae5-f146d0406304\" (UID: \"25c23ffe-4aa3-48f7-aae5-f146d0406304\") "
Oct 30 07:06:23.318553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.318522    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume" pod "82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" (UID: "82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:23.318898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.318872    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume" pod "25c23ffe-4aa3-48f7-aae5-f146d0406304" (UID: "25c23ffe-4aa3-48f7-aae5-f146d0406304"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:23.319252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.319196    6738 scope.go:122] "RemoveContainer" containerID="e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"
Oct 30 07:06:23.320938 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:23.319577    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb\": container with ID starting with e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb not found: ID does not exist" containerID="e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"
Oct 30 07:06:23.320938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.319604    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb"} err="failed to get container status \"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb\": rpc error: code = NotFound desc = could not find container \"e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb\": container with ID starting with e1bad0c73521336d4816d5e8a6300ff120522958e1e59c8e191bb6af9c39c8eb not found: ID does not exist"
Oct 30 07:06:23.320938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.319621    6738 scope.go:122] "RemoveContainer" containerID="caceb77fcc0b930fb5b2236018c638cb1c75fa2fe2a2c8588ec8dc09c3fe177b"
Oct 30 07:06:23.320938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.319848    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52" pod "25c23ffe-4aa3-48f7-aae5-f146d0406304" (UID: "25c23ffe-4aa3-48f7-aae5-f146d0406304"). InnerVolumeSpecName "kube-api-access-nmd52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:23.320938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.320485    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8" pod "82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" (UID: "82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494"). InnerVolumeSpecName "kube-api-access-xnrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:23.418455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.418421    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:23.418455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.418449    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnrv8\" (UniqueName: \"kubernetes.io/projected/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494-kube-api-access-xnrv8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:23.418455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.418459    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c23ffe-4aa3-48f7-aae5-f146d0406304-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:23.418455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.418467    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmd52\" (UniqueName: \"kubernetes.io/projected/25c23ffe-4aa3-48f7-aae5-f146d0406304-kube-api-access-nmd52\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:23.655969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.655883    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:06:23.657693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.657663    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-414c5686-8fc69dfd-8lsnt"]
Oct 30 07:06:23.690590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.690550    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" path="/var/lib/kubelet/pods/82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494/volumes"
Oct 30 07:06:23.716812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.716766    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:06:23.755018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.754984    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:06:23.891362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:23.891330    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:24.005648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.005555    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:24.320023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.319935    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"
Oct 30 07:06:24.320023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.319964    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4" event={"ID":"25c23ffe-4aa3-48f7-aae5-f146d0406304","Type":"ContainerDied","Data":"d5aa146fcdf38060c34fb514ce7a1d58a4a238c195b3aab269527fe1fc4a257b"}
Oct 30 07:06:24.374856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.374601    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:06:24.379392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.377757    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-414c5686-7bb4864d9f-bp2n4"]
Oct 30 07:06:24.987622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:24.987571    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:25.597389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.597352    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:25.689088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.689055    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c23ffe-4aa3-48f7-aae5-f146d0406304" path="/var/lib/kubelet/pods/25c23ffe-4aa3-48f7-aae5-f146d0406304/volumes"
Oct 30 07:06:25.724207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.724174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:25.781195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.781162    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"]
Oct 30 07:06:25.793335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.793276    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-414c5686-kprp5"]
Oct 30 07:06:25.844959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.844856    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:25.933576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.933538    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"]
Oct 30 07:06:25.934132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934106    6738 cpu_manager.go:414] "containerMap: removing container" podUID="25c23ffe-4aa3-48f7-aae5-f146d0406304" containerName="payload-pre-processing"
Oct 30 07:06:25.934132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934128    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c23ffe-4aa3-48f7-aae5-f146d0406304" containerName="payload-pre-processing"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934141    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3241e905-8e8c-473d-b117-bc0754c0638b" containerName="revoke-keys"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934148    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3241e905-8e8c-473d-b117-bc0754c0638b" containerName="revoke-keys"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934176    6738 cpu_manager.go:414] "containerMap: removing container" podUID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" containerName="payload-processing"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934181    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" containerName="payload-processing"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934266    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e06f1f-d7b4-4fc6-9aa0-d7e6a99a2494" containerName="payload-processing"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934280    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c23ffe-4aa3-48f7-aae5-f146d0406304" containerName="payload-pre-processing"
Oct 30 07:06:25.934328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.934287    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3241e905-8e8c-473d-b117-bc0754c0638b" containerName="revoke-keys"
Oct 30 07:06:25.938731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.938711    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:25.954891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.954851    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:06:25.955278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.954918    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:06:25.955278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.955048    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:06:25.955983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.955959    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"]
Oct 30 07:06:25.965576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.965538    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:25.965943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:25.965918    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:06:26.049756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.048859    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.049756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.049001    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.049756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.049142    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpdz\" (UniqueName: \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.078132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.078096    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"]
Oct 30 07:06:26.082062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.082034    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.103514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.103479    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"]
Oct 30 07:06:26.150016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.149974    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.150218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150035    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpdz\" (UniqueName: \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.150218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150083    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.150218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150143    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.150218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150188    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.150493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150454    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5wl\" (UniqueName: \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.150772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.150754    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.152799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.152758    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.174419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.174384    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpdz\" (UniqueName: \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\") pod \"maas-api-revoke-keys-e2e-ait-5b764790-m5mvj\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.251249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.251156    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.251249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.251222    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.251507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.251259    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5wl\" (UniqueName: \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.251947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.251923    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.251947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.251940    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:26.253701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.253682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.274965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.274938    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5wl\" (UniqueName: \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\") pod \"maas-api-revoke-keys-e2e-derive-415d1917-hn8t2\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.396693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.396649    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:26.559338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.559190    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"]
Oct 30 07:06:26.707353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.707324    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"]
Oct 30 07:06:26.709953 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:26.709919    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaab9fbe_523d_49e2_9eff_1978200fc7c7.slice/crio-4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5 WatchSource:0}: Error finding container 4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5: Status 404 returned error can't find the container with id 4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5
Oct 30 07:06:26.800123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:26.800083    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:27.244774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.244738    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:06:27.249691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.249664    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262466    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262508    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262587    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfvp\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262637    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262658    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262722    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262814    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262838    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.262914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.262902    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.274708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.274675    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:06:27.283265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.283230    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2eda2c-openshift-default-dockercfg-bdk2w\""
Oct 30 07:06:27.340804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.339541    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2" event={"ID":"caab9fbe-523d-49e2-9eff-1978200fc7c7","Type":"ContainerStarted","Data":"1efd0495a6642a4a1c958c1a3947afbeea4e4d59be30286562d9dfc630b0f1ea"}
Oct 30 07:06:27.340804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.339614    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2" event={"ID":"caab9fbe-523d-49e2-9eff-1978200fc7c7","Type":"ContainerStarted","Data":"4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.348056    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj" event={"ID":"aee53bfc-351d-496a-b571-4edc46c724d7","Type":"ContainerStarted","Data":"dcf5d44236b5eb98486cbd469b07d991c5d297e011c929619163a21a1a21dde3"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.348126    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj" event={"ID":"aee53bfc-351d-496a-b571-4edc46c724d7","Type":"ContainerStarted","Data":"0c10097b12c6e6b4aab6fa4870d8f0065fde2fd02aaa731acf004f1e26c4e4a1"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.354162    6738 generic.go:356] "Generic (PLEG): container finished" podID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerID="a0acdd37652a9ac2697d52d28ae7a1e0241bc9edc99d5bed40846385c425f6b4" exitCode=0
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.354242    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerDied","Data":"a0acdd37652a9ac2697d52d28ae7a1e0241bc9edc99d5bed40846385c425f6b4"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.354719    6738 scope.go:122] "RemoveContainer" containerID="a0acdd37652a9ac2697d52d28ae7a1e0241bc9edc99d5bed40846385c425f6b4"
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.356458    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.356555    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.356605    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.358119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.356859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:27.363642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.363607    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.363790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.363682    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364000    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364086    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364191    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364239    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364277    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfvp\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364344    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.364549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.364374    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.365344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.365207    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.366545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.365675    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.366545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.365911    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.366545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.366495    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.366896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.366869    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.377214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.377176    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.377966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.377939    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.408129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.408097    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.415341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.410156    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfvp\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.422681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.422620    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2" podStartSLOduration=1.4226035320000001 podStartE2EDuration="1.422603532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:27.421580325 +0000 UTC m=+547.854344218" watchObservedRunningTime="2026-10-30 07:06:27.422603532 +0000 UTC m=+547.855367418"
Oct 30 07:06:27.466760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.466714    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj" podStartSLOduration=2.466701023 podStartE2EDuration="2.466701023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:27.465102761 +0000 UTC m=+547.897866655" watchObservedRunningTime="2026-10-30 07:06:27.466701023 +0000 UTC m=+547.899464916"
Oct 30 07:06:27.567174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.567138    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:27.694427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.694390    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9187a065-b98c-4813-a747-83a3211fb92d" path="/var/lib/kubelet/pods/9187a065-b98c-4813-a747-83a3211fb92d/volumes"
Oct 30 07:06:27.784424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.784279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:27.906280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.906245    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:06:27.913588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.913548    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.913686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.913620    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.913686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.913649    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:27.922420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:27.922389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:28.040725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.040607    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:28.361826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.361734    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerStarted","Data":"3bc2b0db7973a928c1347492b21ff701cf9cd9080a342356ed232ce3e8c48d54"}
Oct 30 07:06:28.363316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.363276    6738 generic.go:356] "Generic (PLEG): container finished" podID="caab9fbe-523d-49e2-9eff-1978200fc7c7" containerID="1efd0495a6642a4a1c958c1a3947afbeea4e4d59be30286562d9dfc630b0f1ea" exitCode=0
Oct 30 07:06:28.363442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.363365    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2" event={"ID":"caab9fbe-523d-49e2-9eff-1978200fc7c7","Type":"ContainerDied","Data":"1efd0495a6642a4a1c958c1a3947afbeea4e4d59be30286562d9dfc630b0f1ea"}
Oct 30 07:06:28.364892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.364869    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerStarted","Data":"c5c71f4e404fbe0905b8ad1c8d9661b0b012fe4930116eff0126863ee69e2e92"}
Oct 30 07:06:28.364993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.364915    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerStarted","Data":"38767940bcd8117b9adb3c7c70a8f599a64c3ced055206aa272fdf5c052f56b9"}
Oct 30 07:06:28.366319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.366273    6738 generic.go:356] "Generic (PLEG): container finished" podID="aee53bfc-351d-496a-b571-4edc46c724d7" containerID="dcf5d44236b5eb98486cbd469b07d991c5d297e011c929619163a21a1a21dde3" exitCode=0
Oct 30 07:06:28.366388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.366317    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj" event={"ID":"aee53bfc-351d-496a-b571-4edc46c724d7","Type":"ContainerDied","Data":"dcf5d44236b5eb98486cbd469b07d991c5d297e011c929619163a21a1a21dde3"}
Oct 30 07:06:28.484316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.472915    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podStartSLOduration=1.472894919 podStartE2EDuration="1.472894919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:28.471799322 +0000 UTC m=+548.904563216" watchObservedRunningTime="2026-10-30 07:06:28.472894919 +0000 UTC m=+548.905658814"
Oct 30 07:06:28.572727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.572650    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:28.617760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.617677    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:28.717834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.717790    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" podUID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" containerName="payload-processing" containerID="cri-o://06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83" gracePeriod=25
Oct 30 07:06:28.755935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.755897    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerName="payload-pre-processing" containerID="cri-o://4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b" gracePeriod=25
Oct 30 07:06:28.865468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.865439    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:28.866933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.866903    6738 patch_prober.go:29] interesting pod/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 07:06:28.867037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:28.866970    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 07:06:29.056278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.056251    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:06:29.104047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.104020    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:06:29.109185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.109160    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\" (UniqueName: \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\") pod \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") "
Oct 30 07:06:29.109278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.109206    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\") pod \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\" (UID: \"06657cd5-fe4a-4b28-b8cf-e8a545adaa61\") "
Oct 30 07:06:29.109610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.109589    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume" pod "06657cd5-fe4a-4b28-b8cf-e8a545adaa61" (UID: "06657cd5-fe4a-4b28-b8cf-e8a545adaa61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:29.111369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.111344    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt" pod "06657cd5-fe4a-4b28-b8cf-e8a545adaa61" (UID: "06657cd5-fe4a-4b28-b8cf-e8a545adaa61"). InnerVolumeSpecName "kube-api-access-4lnwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:29.210390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.210355    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\" (UniqueName: \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\") pod \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") "
Oct 30 07:06:29.210590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.210457    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\") pod \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\" (UID: \"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7\") "
Oct 30 07:06:29.210669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.210650    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lnwt\" (UniqueName: \"kubernetes.io/projected/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-kube-api-access-4lnwt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.210669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.210662    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06657cd5-fe4a-4b28-b8cf-e8a545adaa61-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.210904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.210878    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume" pod "e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" (UID: "e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:29.212577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.212548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5" pod "e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" (UID: "e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7"). InnerVolumeSpecName "kube-api-access-xg9s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:29.311805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.311770    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9s5\" (UniqueName: \"kubernetes.io/projected/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-kube-api-access-xg9s5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.311805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.311800    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.319160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.319124    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:06:29.401346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.401306    6738 generic.go:356] "Generic (PLEG): container finished" podID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerID="4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b" exitCode=0
Oct 30 07:06:29.401820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.401364    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"
Oct 30 07:06:29.401820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.401367    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" event={"ID":"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7","Type":"ContainerDied","Data":"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"}
Oct 30 07:06:29.401820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.401426    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" event={"ID":"e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7","Type":"ContainerDied","Data":"a5511424a5941ce29cf7e82f0e26099daf533e5ba3b906048852cfac8053043f"}
Oct 30 07:06:29.401820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.401461    6738 scope.go:122] "RemoveContainer" containerID="4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"
Oct 30 07:06:29.403184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.403159    6738 generic.go:356] "Generic (PLEG): container finished" podID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" containerID="06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83" exitCode=0
Oct 30 07:06:29.403328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.403218    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"
Oct 30 07:06:29.403328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.403250    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" event={"ID":"06657cd5-fe4a-4b28-b8cf-e8a545adaa61","Type":"ContainerDied","Data":"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"}
Oct 30 07:06:29.403328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.403318    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz" event={"ID":"06657cd5-fe4a-4b28-b8cf-e8a545adaa61","Type":"ContainerDied","Data":"895d8c3186776ba34b18432a97d558aedc7f72a51c8bb8acbcca474e3bc07e5f"}
Oct 30 07:06:29.404217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.403664    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:29.405058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.405022    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:29.409775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.409391    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:06:29.426971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.426940    6738 scope.go:122] "RemoveContainer" containerID="4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"
Oct 30 07:06:29.427560 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:29.427517    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b\": container with ID starting with 4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b not found: ID does not exist" containerID="4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"
Oct 30 07:06:29.427692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.427574    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b"} err="failed to get container status \"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b\": rpc error: code = NotFound desc = could not find container \"4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b\": container with ID starting with 4fb21b47b8e51a1a790b141225e32b335f1031783e86903a8c255304c086ac8b not found: ID does not exist"
Oct 30 07:06:29.427692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.427617    6738 scope.go:122] "RemoveContainer" containerID="06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"
Oct 30 07:06:29.447282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.447259    6738 scope.go:122] "RemoveContainer" containerID="06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"
Oct 30 07:06:29.448113 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:29.448083    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83\": container with ID starting with 06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83 not found: ID does not exist" containerID="06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"
Oct 30 07:06:29.448269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.448241    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83"} err="failed to get container status \"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83\": rpc error: code = NotFound desc = could not find container \"06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83\": container with ID starting with 06fd903b0b44b4e6d00a607f1db49f305195c93fdea5b5aee799804dd3cfbc83 not found: ID does not exist"
Oct 30 07:06:29.505453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.501224    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:06:29.505453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.501313    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78"]
Oct 30 07:06:29.534114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.531098    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:29.638863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.638830    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:06:29.643480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.643447    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9eddc5d-77454fcf76-phzzz"]
Oct 30 07:06:29.649075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.649048    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:29.656881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.656853    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:29.659129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.659104    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:29.700711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.700622    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" path="/var/lib/kubelet/pods/06657cd5-fe4a-4b28-b8cf-e8a545adaa61/volumes"
Oct 30 07:06:29.701158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.701141    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" path="/var/lib/kubelet/pods/e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7/volumes"
Oct 30 07:06:29.723407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723370    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\") pod \"aee53bfc-351d-496a-b571-4edc46c724d7\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") "
Oct 30 07:06:29.723564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723429    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\") pod \"caab9fbe-523d-49e2-9eff-1978200fc7c7\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") "
Oct 30 07:06:29.723564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723544    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\" (UniqueName: \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\") pod \"aee53bfc-351d-496a-b571-4edc46c724d7\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") "
Oct 30 07:06:29.723672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723600    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\") pod \"aee53bfc-351d-496a-b571-4edc46c724d7\" (UID: \"aee53bfc-351d-496a-b571-4edc46c724d7\") "
Oct 30 07:06:29.723672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723658    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\" (UniqueName: \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\") pod \"caab9fbe-523d-49e2-9eff-1978200fc7c7\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") "
Oct 30 07:06:29.723771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723704    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\") pod \"caab9fbe-523d-49e2-9eff-1978200fc7c7\" (UID: \"caab9fbe-523d-49e2-9eff-1978200fc7c7\") "
Oct 30 07:06:29.723820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.723775    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca" pod "aee53bfc-351d-496a-b571-4edc46c724d7" (UID: "aee53bfc-351d-496a-b571-4edc46c724d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:29.724557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.724139    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.725368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.725342    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca" pod "caab9fbe-523d-49e2-9eff-1978200fc7c7" (UID: "caab9fbe-523d-49e2-9eff-1978200fc7c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:29.726581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.726552    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca" pod "aee53bfc-351d-496a-b571-4edc46c724d7" (UID: "aee53bfc-351d-496a-b571-4edc46c724d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:29.726581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.726568    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz" pod "aee53bfc-351d-496a-b571-4edc46c724d7" (UID: "aee53bfc-351d-496a-b571-4edc46c724d7"). InnerVolumeSpecName "kube-api-access-9dpdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:29.727258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.727235    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl" pod "caab9fbe-523d-49e2-9eff-1978200fc7c7" (UID: "caab9fbe-523d-49e2-9eff-1978200fc7c7"). InnerVolumeSpecName "kube-api-access-tm5wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:29.727578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.727560    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca" pod "caab9fbe-523d-49e2-9eff-1978200fc7c7" (UID: "caab9fbe-523d-49e2-9eff-1978200fc7c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:29.828102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.827952    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpdz\" (UniqueName: \"kubernetes.io/projected/aee53bfc-351d-496a-b571-4edc46c724d7-kube-api-access-9dpdz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.828102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.827990    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee53bfc-351d-496a-b571-4edc46c724d7-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.828102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.828006    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5wl\" (UniqueName: \"kubernetes.io/projected/caab9fbe-523d-49e2-9eff-1978200fc7c7-kube-api-access-tm5wl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.828102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.828021    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.828102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.828036    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caab9fbe-523d-49e2-9eff-1978200fc7c7-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:29.865514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.865472    6738 patch_prober.go:29] interesting pod/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 07:06:29.865693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:29.865535    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 07:06:30.057156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.057042    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9eddc5d-b64b5bc9f-fdb78" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:50052\" within 1s: context deadline exceeded"
Oct 30 07:06:30.057380 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:30.057158    6738 logging.go:55] [core] [Channel #67 SubChannel #70]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: operation was canceled"
Oct 30 07:06:30.063124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.063097    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:30.195316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.194380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:30.410102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.410068    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj" event={"ID":"aee53bfc-351d-496a-b571-4edc46c724d7","Type":"ContainerDied","Data":"0c10097b12c6e6b4aab6fa4870d8f0065fde2fd02aaa731acf004f1e26c4e4a1"}
Oct 30 07:06:30.410582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.410126    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c10097b12c6e6b4aab6fa4870d8f0065fde2fd02aaa731acf004f1e26c4e4a1"
Oct 30 07:06:30.410582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.410083    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"
Oct 30 07:06:30.413395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.413366    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2" event={"ID":"caab9fbe-523d-49e2-9eff-1978200fc7c7","Type":"ContainerDied","Data":"4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5"}
Oct 30 07:06:30.413528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.413408    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"
Oct 30 07:06:30.413528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.413414    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3cd70f5928db01910786ce24f74a7428e481401265f729af2127f6d270f9c5"
Oct 30 07:06:30.865932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.865850    6738 patch_prober.go:29] interesting pod/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 07:06:30.865932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.865905    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 07:06:30.965757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:30.965724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:31.094188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:31.094145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:31.866378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:31.866336    6738 patch_prober.go:29] interesting pod/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 07:06:31.866762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:31.866407    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 07:06:31.995054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:31.995012    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:32.757186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:32.757152    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:32.866001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:32.865961    6738 patch_prober.go:29] interesting pod/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 07:06:32.866186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:32.866024    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 07:06:32.894911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:32.894877    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:33.013364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.011311    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"]
Oct 30 07:06:33.045317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.044516    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9eddc5d-pnrkq"]
Oct 30 07:06:33.131389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.131348    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:33.264549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.264456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:33.688501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.688471    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21f1bff-98a0-4c64-89dd-b10ea95dcad4" path="/var/lib/kubelet/pods/c21f1bff-98a0-4c64-89dd-b10ea95dcad4/volumes"
Oct 30 07:06:33.870180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.870151    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:33.871054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:33.871036    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:34.089385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.089271    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:34.320434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.320386    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" podUID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" containerName="payload-processing" containerID="cri-o://badb5e6b4e1e75e99925ac923cc887cc6dad497d63f9e563ac453821fd795567" gracePeriod=25
Oct 30 07:06:34.410326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.410272    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" podUID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" containerName="payload-pre-processing" containerID="cri-o://52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657" gracePeriod=25
Oct 30 07:06:34.443757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.443724    6738 generic.go:356] "Generic (PLEG): container finished" podID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" containerID="badb5e6b4e1e75e99925ac923cc887cc6dad497d63f9e563ac453821fd795567" exitCode=0
Oct 30 07:06:34.443908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.443793    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" event={"ID":"26d5bd40-a0c1-4b54-8f4c-105ab10fff54","Type":"ContainerDied","Data":"badb5e6b4e1e75e99925ac923cc887cc6dad497d63f9e563ac453821fd795567"}
Oct 30 07:06:34.612180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.612143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:34.705125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.705095    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:06:34.708498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.708474    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:06:34.735558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.735507    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:34.778448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778411    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\" (UniqueName: \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\") pod \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") "
Oct 30 07:06:34.778621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778457    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\" (UniqueName: \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\") pod \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") "
Oct 30 07:06:34.778621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778519    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\") pod \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\" (UID: \"8aba1948-31a6-4fa0-b40f-65d813fe5e1c\") "
Oct 30 07:06:34.778621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778558    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\") pod \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\" (UID: \"26d5bd40-a0c1-4b54-8f4c-105ab10fff54\") "
Oct 30 07:06:34.778977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778920    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume" pod "8aba1948-31a6-4fa0-b40f-65d813fe5e1c" (UID: "8aba1948-31a6-4fa0-b40f-65d813fe5e1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:34.778977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.778921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume" pod "26d5bd40-a0c1-4b54-8f4c-105ab10fff54" (UID: "26d5bd40-a0c1-4b54-8f4c-105ab10fff54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:34.781220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.781197    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7" pod "8aba1948-31a6-4fa0-b40f-65d813fe5e1c" (UID: "8aba1948-31a6-4fa0-b40f-65d813fe5e1c"). InnerVolumeSpecName "kube-api-access-vrvt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:34.781367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.781225    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf" pod "26d5bd40-a0c1-4b54-8f4c-105ab10fff54" (UID: "26d5bd40-a0c1-4b54-8f4c-105ab10fff54"). InnerVolumeSpecName "kube-api-access-kvlrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:34.848203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.848165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:34.880338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.880284    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:34.880338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.880341    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:34.880551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.880358    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvlrf\" (UniqueName: \"kubernetes.io/projected/26d5bd40-a0c1-4b54-8f4c-105ab10fff54-kube-api-access-kvlrf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:34.880551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:34.880373    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvt7\" (UniqueName: \"kubernetes.io/projected/8aba1948-31a6-4fa0-b40f-65d813fe5e1c-kube-api-access-vrvt7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:35.250197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.250159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:35.370409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.370372    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:35.388937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.388898    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"]
Oct 30 07:06:35.401210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.401172    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1a08662-qth4x"]
Oct 30 07:06:35.450560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.450528    6738 generic.go:356] "Generic (PLEG): container finished" podID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" containerID="52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657" exitCode=0
Oct 30 07:06:35.450762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.450575    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"
Oct 30 07:06:35.450762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.450619    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" event={"ID":"8aba1948-31a6-4fa0-b40f-65d813fe5e1c","Type":"ContainerDied","Data":"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"}
Oct 30 07:06:35.450762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.450672    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n" event={"ID":"8aba1948-31a6-4fa0-b40f-65d813fe5e1c","Type":"ContainerDied","Data":"e2341c5537a2cb2d2947654477745ebb4e2e9a3b3193202b930e7d923a615c52"}
Oct 30 07:06:35.450762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.450702    6738 scope.go:122] "RemoveContainer" containerID="52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"
Oct 30 07:06:35.452225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.452207    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"
Oct 30 07:06:35.452327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.452239    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r" event={"ID":"26d5bd40-a0c1-4b54-8f4c-105ab10fff54","Type":"ContainerDied","Data":"ec6c8a7b9c63c6cf10b71c5ded7618dfccef67ffc2e45543578eee0ce249ce6b"}
Oct 30 07:06:35.459889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.459867    6738 scope.go:122] "RemoveContainer" containerID="52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"
Oct 30 07:06:35.460245 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:35.460221    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657\": container with ID starting with 52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657 not found: ID does not exist" containerID="52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"
Oct 30 07:06:35.460325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.460259    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657"} err="failed to get container status \"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657\": rpc error: code = NotFound desc = could not find container \"52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657\": container with ID starting with 52092693d8186b1886a0e966824644e5e6d13102643247d187bc0fa9ad7c6657 not found: ID does not exist"
Oct 30 07:06:35.460325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.460288    6738 scope.go:122] "RemoveContainer" containerID="badb5e6b4e1e75e99925ac923cc887cc6dad497d63f9e563ac453821fd795567"
Oct 30 07:06:35.493305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.493258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:35.502535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.502455    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:06:35.509602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.509574    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1a08662-7c7fbf748d-s4g5r"]
Oct 30 07:06:35.544761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.544722    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:06:35.549854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.549826    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1a08662-6558b5746c-h9p9n"]
Oct 30 07:06:35.695279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.695244    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" path="/var/lib/kubelet/pods/26d5bd40-a0c1-4b54-8f4c-105ab10fff54/volumes"
Oct 30 07:06:35.696069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.696045    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3241e905-8e8c-473d-b117-bc0754c0638b" path="/var/lib/kubelet/pods/3241e905-8e8c-473d-b117-bc0754c0638b/volumes"
Oct 30 07:06:35.696700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.696675    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" path="/var/lib/kubelet/pods/8aba1948-31a6-4fa0-b40f-65d813fe5e1c/volumes"
Oct 30 07:06:35.835959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.835874    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:35.956048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:35.956017    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:36.472832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:36.472799    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:36.595230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:36.595194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:36.722884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:36.722799    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:37.076280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:37.076191    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:37.457903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:37.457869    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:37.518120 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:37.518081    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:06:37.568514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:37.568424    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:37.952796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:37.952752    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:37.988514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:37.988481    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:06:38.080341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.080289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:38.260081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.256801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:38.404444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404409    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:06:38.404853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404829    6738 cpu_manager.go:414] "containerMap: removing container" podUID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" containerName="payload-processing"
Oct 30 07:06:38.404853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404846    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" containerName="payload-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404874    6738 cpu_manager.go:414] "containerMap: removing container" podUID="caab9fbe-523d-49e2-9eff-1978200fc7c7" containerName="revoke-keys"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404879    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caab9fbe-523d-49e2-9eff-1978200fc7c7" containerName="revoke-keys"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404887    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" containerName="payload-pre-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404892    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" containerName="payload-pre-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404912    6738 cpu_manager.go:414] "containerMap: removing container" podUID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" containerName="payload-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404916    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" containerName="payload-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404924    6738 cpu_manager.go:414] "containerMap: removing container" podUID="aee53bfc-351d-496a-b571-4edc46c724d7" containerName="revoke-keys"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404932    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee53bfc-351d-496a-b571-4edc46c724d7" containerName="revoke-keys"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404955    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerName="payload-pre-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.404960    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerName="payload-pre-processing"
Oct 30 07:06:38.405019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405022    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="caab9fbe-523d-49e2-9eff-1978200fc7c7" containerName="revoke-keys"
Oct 30 07:06:38.405398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405030    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aba1948-31a6-4fa0-b40f-65d813fe5e1c" containerName="payload-pre-processing"
Oct 30 07:06:38.405398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405040    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d5bd40-a0c1-4b54-8f4c-105ab10fff54" containerName="payload-processing"
Oct 30 07:06:38.405398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405047    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee53bfc-351d-496a-b571-4edc46c724d7" containerName="revoke-keys"
Oct 30 07:06:38.405398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405054    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20eeb1a-0d19-44f7-a08f-edcea8bdb8b7" containerName="payload-pre-processing"
Oct 30 07:06:38.405398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.405060    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="06657cd5-fe4a-4b28-b8cf-e8a545adaa61" containerName="payload-processing"
Oct 30 07:06:38.409878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.409846    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.432553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.432511    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2bab4c-openshift-default-dockercfg-gqrfc\""
Oct 30 07:06:38.434812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.434782    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:06:38.468310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.468268    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:38.521273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521186    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521242    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wwz\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521383    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521457    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521590    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521634    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521684    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.521763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.521716    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.622672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622636    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.622846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622695    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wwz\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.622846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622739    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.622846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622771    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.622846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622832    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622877    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622915    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.622951    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623001    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623339    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623380    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623441    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.623582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.623563    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.625258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.625228    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.625500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.625481    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.664166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.664127    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.671610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.671542    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wwz\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.724311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.724252    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:38.748481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.748441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:38.872606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:38.872570    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:39.051611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.051579    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:06:39.052924 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:39.052898    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cabdbc_f6f2_4995_8a38_2501eb972c3a.slice/crio-6fb335c0dae4d218d491074c24151e67f72e28cfde8e29dfd81f6fe00d152abb WatchSource:0}: Error finding container 6fb335c0dae4d218d491074c24151e67f72e28cfde8e29dfd81f6fe00d152abb: Status 404 returned error can't find the container with id 6fb335c0dae4d218d491074c24151e67f72e28cfde8e29dfd81f6fe00d152abb
Oct 30 07:06:39.055333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.055259    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:39.055419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.055361    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:39.055419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.055398    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:39.460865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.460831    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:39.477438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.477388    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerStarted","Data":"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"}
Oct 30 07:06:39.477593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.477455    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerStarted","Data":"6fb335c0dae4d218d491074c24151e67f72e28cfde8e29dfd81f6fe00d152abb"}
Oct 30 07:06:39.525482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.525417    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podStartSLOduration=1.525400706 podStartE2EDuration="1.525400706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:39.524732429 +0000 UTC m=+559.957496321" watchObservedRunningTime="2026-10-30 07:06:39.525400706 +0000 UTC m=+559.958164590"
Oct 30 07:06:39.725078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.724995    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:39.941227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:39.941188    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:40.449794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.449747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:40.669669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.669633    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:40.718832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.718803    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"]
Oct 30 07:06:40.729482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.729452    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:40.730631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.730608    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b764790-m5mvj"]
Oct 30 07:06:40.788712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.788674    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:40.919948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:40.919861    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:41.042269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.042222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:41.119925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.119889    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"]
Oct 30 07:06:41.125042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.125012    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-415d1917-hn8t2"]
Oct 30 07:06:41.154995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.154952    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:41.275499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.275407    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:06:41.275693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.275668    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy" containerID="cri-o://3bc2b0db7973a928c1347492b21ff701cf9cd9080a342356ed232ce3e8c48d54" gracePeriod=30
Oct 30 07:06:41.325592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.325558    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:41.469963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.469922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:41.473705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.473670    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" podUID="893d3ca1-e493-434d-b95d-b0ce544086a8" containerName="payload-pre-processing" containerID="cri-o://c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1" gracePeriod=25
Oct 30 07:06:41.488472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.488443    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:41.489596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.489442    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:06:41.689546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.689511    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee53bfc-351d-496a-b571-4edc46c724d7" path="/var/lib/kubelet/pods/aee53bfc-351d-496a-b571-4edc46c724d7/volumes"
Oct 30 07:06:41.690107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.690087    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caab9fbe-523d-49e2-9eff-1978200fc7c7" path="/var/lib/kubelet/pods/caab9fbe-523d-49e2-9eff-1978200fc7c7/volumes"
Oct 30 07:06:41.755464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.755428    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:41.804012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.803977    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\" (UniqueName: \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\") pod \"893d3ca1-e493-434d-b95d-b0ce544086a8\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") "
Oct 30 07:06:41.804199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.804096    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\") pod \"893d3ca1-e493-434d-b95d-b0ce544086a8\" (UID: \"893d3ca1-e493-434d-b95d-b0ce544086a8\") "
Oct 30 07:06:41.804799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.804465    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume" pod "893d3ca1-e493-434d-b95d-b0ce544086a8" (UID: "893d3ca1-e493-434d-b95d-b0ce544086a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:41.804799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.804566    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893d3ca1-e493-434d-b95d-b0ce544086a8-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:41.806834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.806812    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv" pod "893d3ca1-e493-434d-b95d-b0ce544086a8" (UID: "893d3ca1-e493-434d-b95d-b0ce544086a8"). InnerVolumeSpecName "kube-api-access-wcsvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:41.908574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:41.908527    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsvv\" (UniqueName: \"kubernetes.io/projected/893d3ca1-e493-434d-b95d-b0ce544086a8-kube-api-access-wcsvv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:42.189379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.186954    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:42.329377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.329334    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:42.498002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.497914    6738 generic.go:356] "Generic (PLEG): container finished" podID="893d3ca1-e493-434d-b95d-b0ce544086a8" containerID="c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1" exitCode=0
Oct 30 07:06:42.498002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.497964    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"
Oct 30 07:06:42.498199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.498001    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" event={"ID":"893d3ca1-e493-434d-b95d-b0ce544086a8","Type":"ContainerDied","Data":"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"}
Oct 30 07:06:42.498199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.498067    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt" event={"ID":"893d3ca1-e493-434d-b95d-b0ce544086a8","Type":"ContainerDied","Data":"ce2d3f79b239fe74dc9871ef3bbe1d7a78f74dd9a02fecd30a30e05d483c0a2e"}
Oct 30 07:06:42.498199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.498100    6738 scope.go:122] "RemoveContainer" containerID="c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"
Oct 30 07:06:42.508516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.508485    6738 scope.go:122] "RemoveContainer" containerID="c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"
Oct 30 07:06:42.508889 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:42.508864    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1\": container with ID starting with c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1 not found: ID does not exist" containerID="c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"
Oct 30 07:06:42.508956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.508901    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1"} err="failed to get container status \"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1\": rpc error: code = NotFound desc = could not find container \"c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1\": container with ID starting with c960a0c16f8a60c5a5a1915fb80c652b0abcd11bdd8639516dc17c0fa4b2f0b1 not found: ID does not exist"
Oct 30 07:06:42.553130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.553093    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:42.557767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.557733    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-415d1917-74dc5d9fc5-mnljt"]
Oct 30 07:06:42.932437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.932399    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:42.993613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:42.993428    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" podUID="dc48d84e-e105-4e63-b587-33769433075b" containerName="payload-processing" containerID="cri-o://fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736" gracePeriod=25
Oct 30 07:06:43.096321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.092287    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:06:43.096321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.093609    6738 cpu_manager.go:414] "containerMap: removing container" podUID="893d3ca1-e493-434d-b95d-b0ce544086a8" containerName="payload-pre-processing"
Oct 30 07:06:43.096321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.093632    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893d3ca1-e493-434d-b95d-b0ce544086a8" containerName="payload-pre-processing"
Oct 30 07:06:43.096321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.093764    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="893d3ca1-e493-434d-b95d-b0ce544086a8" containerName="payload-pre-processing"
Oct 30 07:06:43.102324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.102084    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:43.107241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.107212    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.110044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.110016    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:06:43.204448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.204417    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-06b26e-openshift-default-dockercfg-t2k6l\""
Oct 30 07:06:43.219554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.219515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228280    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228349    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228388    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228470    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228510    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228525    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7s62\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228554    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228569    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.230332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.228590    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.304155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.304130    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:06:43.329421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329424    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329447    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329494    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329552    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329575    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7s62\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329685    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329725    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.329879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329768    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.330024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329946    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.330024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.329985    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.330110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.330063    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.330200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.330181    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.330460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.330444    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.331704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.331679    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.332006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.331989    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.354534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.354496    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7s62\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.355548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.355494    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.397149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.396007    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:43.432315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.429579    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:43.432315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.430354    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\" (UniqueName: \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\") pod \"dc48d84e-e105-4e63-b587-33769433075b\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") "
Oct 30 07:06:43.432315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.430431    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\") pod \"dc48d84e-e105-4e63-b587-33769433075b\" (UID: \"dc48d84e-e105-4e63-b587-33769433075b\") "
Oct 30 07:06:43.434564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.434531    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl" pod "dc48d84e-e105-4e63-b587-33769433075b" (UID: "dc48d84e-e105-4e63-b587-33769433075b"). InnerVolumeSpecName "kube-api-access-8gvkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:43.435671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.435622    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume" pod "dc48d84e-e105-4e63-b587-33769433075b" (UID: "dc48d84e-e105-4e63-b587-33769433075b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:43.516635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.516504    6738 generic.go:356] "Generic (PLEG): container finished" podID="dc48d84e-e105-4e63-b587-33769433075b" containerID="fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736" exitCode=0
Oct 30 07:06:43.519878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.516583    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" event={"ID":"dc48d84e-e105-4e63-b587-33769433075b","Type":"ContainerDied","Data":"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"}
Oct 30 07:06:43.519878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.516902    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m" event={"ID":"dc48d84e-e105-4e63-b587-33769433075b","Type":"ContainerDied","Data":"cb1de29f4c2ee55ca4c237259cdbd612b920c635d048a259013a1a4a1013d449"}
Oct 30 07:06:43.519878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.516937    6738 scope.go:122] "RemoveContainer" containerID="fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"
Oct 30 07:06:43.519878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.516621    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"
Oct 30 07:06:43.533279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.533255    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvkl\" (UniqueName: \"kubernetes.io/projected/dc48d84e-e105-4e63-b587-33769433075b-kube-api-access-8gvkl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:43.533517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.533501    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48d84e-e105-4e63-b587-33769433075b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:43.534780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.534485    6738 scope.go:122] "RemoveContainer" containerID="fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"
Oct 30 07:06:43.535453 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:06:43.535179    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736\": container with ID starting with fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736 not found: ID does not exist" containerID="fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"
Oct 30 07:06:43.535453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.535220    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736"} err="failed to get container status \"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736\": rpc error: code = NotFound desc = could not find container \"fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736\": container with ID starting with fa56164145b665efa6bb978953a9e2a975627c49058c30cc79072000bb251736 not found: ID does not exist"
Oct 30 07:06:43.586332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.582883    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:06:43.597907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.597867    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b764790-7c4f555574-h8f5m"]
Oct 30 07:06:43.700216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.700178    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893d3ca1-e493-434d-b95d-b0ce544086a8" path="/var/lib/kubelet/pods/893d3ca1-e493-434d-b95d-b0ce544086a8/volumes"
Oct 30 07:06:43.700710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.700688    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc48d84e-e105-4e63-b587-33769433075b" path="/var/lib/kubelet/pods/dc48d84e-e105-4e63-b587-33769433075b/volumes"
Oct 30 07:06:43.794584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.792344    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:06:43.795017 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:43.794978    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94efc78_0fa1_454c_b59a_e65491208bfc.slice/crio-ed19b9ea98e556cfc01bcc180b6b36633508da857644ed4fb08ba4733364f2db WatchSource:0}: Error finding container ed19b9ea98e556cfc01bcc180b6b36633508da857644ed4fb08ba4733364f2db: Status 404 returned error can't find the container with id ed19b9ea98e556cfc01bcc180b6b36633508da857644ed4fb08ba4733364f2db
Oct 30 07:06:43.798168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.798128    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:43.798243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.798223    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:43.798316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:43.798273    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:44.542230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.542195    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerStarted","Data":"1f936c455a6871cf6b997e8679e50bfa95febda20c6a79445b05a872fb1ba7d7"}
Oct 30 07:06:44.542775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.542253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerStarted","Data":"ed19b9ea98e556cfc01bcc180b6b36633508da857644ed4fb08ba4733364f2db"}
Oct 30 07:06:44.654261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.654222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:44.794974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.794872    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:44.812665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.812402    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podStartSLOduration=1.8123821169999998 podStartE2EDuration="1.812382117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:44.599752235 +0000 UTC m=+565.032516193" watchObservedRunningTime="2026-10-30 07:06:44.812382117 +0000 UTC m=+565.245146012"
Oct 30 07:06:44.813668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.813638    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 07:06:44.814207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.814183    6738 cpu_manager.go:414] "containerMap: removing container" podUID="dc48d84e-e105-4e63-b587-33769433075b" containerName="payload-processing"
Oct 30 07:06:44.814207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.814206    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc48d84e-e105-4e63-b587-33769433075b" containerName="payload-processing"
Oct 30 07:06:44.814446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.814383    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc48d84e-e105-4e63-b587-33769433075b" containerName="payload-processing"
Oct 30 07:06:44.817228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.817207    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.835855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.835824    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 07:06:44.837601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.837574    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2bab4c-dockercfg-bnm7k\""
Oct 30 07:06:44.848969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.848940    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6k2m\" (UniqueName: \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.849129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.848989    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.911314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.902630    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2bab4c\""
Oct 30 07:06:44.950805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.950764    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6k2m\" (UniqueName: \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.951042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.951030    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.952014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.951994    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:44.995386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:44.995267    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6k2m\" (UniqueName: \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\") pod \"payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:45.130449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.130412    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:45.441575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.440371    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:45.456281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.456236    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:45.501074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.501046    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 07:06:45.503049 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:45.503009    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96d55e9_2df2_42ab_b495_e89dfeff776a.slice/crio-1265f19b9850ba138388f628a8a111d39a1a48399049d4c4e110b174e8ac977b WatchSource:0}: Error finding container 1265f19b9850ba138388f628a8a111d39a1a48399049d4c4e110b174e8ac977b: Status 404 returned error can't find the container with id 1265f19b9850ba138388f628a8a111d39a1a48399049d4c4e110b174e8ac977b
Oct 30 07:06:45.550584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.550560    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" event={"ID":"e96d55e9-2df2-42ab-b495-e89dfeff776a","Type":"ContainerStarted","Data":"1265f19b9850ba138388f628a8a111d39a1a48399049d4c4e110b174e8ac977b"}
Oct 30 07:06:45.550942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.550615    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:45.551606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.551584    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:06:45.766224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:45.766148    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:46.557138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.557058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" event={"ID":"e96d55e9-2df2-42ab-b495-e89dfeff776a","Type":"ContainerStarted","Data":"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"}
Oct 30 07:06:46.557690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.557672    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:46.559341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.559309    6738 generic.go:356] "Generic (PLEG): container finished" podID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerID="3bc2b0db7973a928c1347492b21ff701cf9cd9080a342356ed232ce3e8c48d54" exitCode=0
Oct 30 07:06:46.559487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.559412    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerDied","Data":"3bc2b0db7973a928c1347492b21ff701cf9cd9080a342356ed232ce3e8c48d54"}
Oct 30 07:06:46.559487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.559482    6738 scope.go:122] "RemoveContainer" containerID="a0acdd37652a9ac2697d52d28ae7a1e0241bc9edc99d5bed40846385c425f6b4"
Oct 30 07:06:46.605140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.605112    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:46.618404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.618338    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" podStartSLOduration=2.618320014 podStartE2EDuration="2.618320014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:46.617996896 +0000 UTC m=+567.050760789" watchObservedRunningTime="2026-10-30 07:06:46.618320014 +0000 UTC m=+567.051083908"
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688528    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688608    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688640    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688715    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688748    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688805    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688903    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.688970    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\") pod \"8746406c-0cb4-4a13-9cd8-a278b476dc12\" (UID: \"8746406c-0cb4-4a13-9cd8-a278b476dc12\") "
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.693502    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.693806    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.693994    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:06:46.694320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.694161    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:06:46.700494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.695273    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:06:46.700494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.697724    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:06:46.700494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.697834    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:06:46.700494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.698313    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "kube-api-access-mhtkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:46.700494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.699622    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token" pod "8746406c-0cb4-4a13-9cd8-a278b476dc12" (UID: "8746406c-0cb4-4a13-9cd8-a278b476dc12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.782825    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790829    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790864    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790896    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790911    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790925    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8746406c-0cb4-4a13-9cd8-a278b476dc12-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790938    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790966    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtkf\" (UniqueName: \"kubernetes.io/projected/8746406c-0cb4-4a13-9cd8-a278b476dc12-kube-api-access-mhtkf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790981    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8746406c-0cb4-4a13-9cd8-a278b476dc12-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.790995    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8746406c-0cb4-4a13-9cd8-a278b476dc12-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:06:46.907887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:46.907856    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:47.410725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.410686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:47.554338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.553855    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:47.577967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.577937    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"
Oct 30 07:06:47.578498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.577959    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw" event={"ID":"8746406c-0cb4-4a13-9cd8-a278b476dc12","Type":"ContainerDied","Data":"894a0ed21c495f856f12aa405bc09e5ecd917d094d5728f65ee7607541f5c92c"}
Oct 30 07:06:47.578498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.578065    6738 scope.go:122] "RemoveContainer" containerID="3bc2b0db7973a928c1347492b21ff701cf9cd9080a342356ed232ce3e8c48d54"
Oct 30 07:06:47.648247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.648212    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:06:47.664673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.664584    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-415d1917-openshift-default-84dffd97b5-4q2gw"]
Oct 30 07:06:47.689958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:47.689919    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" path="/var/lib/kubelet/pods/8746406c-0cb4-4a13-9cd8-a278b476dc12/volumes"
Oct 30 07:06:48.117397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:48.117287    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:48.243743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:48.243581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:48.358521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:48.358482    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:50.214764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.214726    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:50.694148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.694106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:50.816521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.816483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:50.867641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.867613    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 07:06:50.868106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.868088    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.868197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.868108    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.868267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.868246    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.870422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.870401    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:50.887841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.887812    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4fe2ed\""
Oct 30 07:06:50.901771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.901738    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 07:06:50.962370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.958780    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726jh\" (UniqueName: \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:50.962370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.958845    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4fe2ed-dockercfg-ftvr5\""
Oct 30 07:06:50.962370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.959027    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:50.966257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.966213    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 07:06:50.973341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.966924    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.973341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.971419    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.973341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.971628    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8746406c-0cb4-4a13-9cd8-a278b476dc12" containerName="istio-proxy"
Oct 30 07:06:50.974508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.974476    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:50.993025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:50.992244    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 07:06:51.005778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.005729    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:51.065380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.062000    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.065380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.062074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.065380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.062207    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszhs\" (UniqueName: \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.065380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.062246    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726jh\" (UniqueName: \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.067353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.067322    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.094591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.094554    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-726jh\" (UniqueName: \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\") pod \"payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.163209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.163170    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.163428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.163259    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszhs\" (UniqueName: \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.163936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.163908    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.187007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.186975    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszhs\" (UniqueName: \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\") pod \"payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.224318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.224216    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.289139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.289106    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:51.527195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.527166    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 07:06:51.527873 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:51.527849    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3264b4cb_dde2_45f2_805b_dca581cdfe56.slice/crio-af324aac0de14fb74c6b352c78b0da4d622841b1bccaf1ef483c687e7d343470 WatchSource:0}: Error finding container af324aac0de14fb74c6b352c78b0da4d622841b1bccaf1ef483c687e7d343470: Status 404 returned error can't find the container with id af324aac0de14fb74c6b352c78b0da4d622841b1bccaf1ef483c687e7d343470
Oct 30 07:06:51.605842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.605767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:51.606830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.606804    6738 generic.go:356] "Generic (PLEG): container finished" podID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerID="c5c71f4e404fbe0905b8ad1c8d9661b0b012fe4930116eff0126863ee69e2e92" exitCode=0
Oct 30 07:06:51.606964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.606884    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerDied","Data":"c5c71f4e404fbe0905b8ad1c8d9661b0b012fe4930116eff0126863ee69e2e92"}
Oct 30 07:06:51.607412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.607390    6738 scope.go:122] "RemoveContainer" containerID="c5c71f4e404fbe0905b8ad1c8d9661b0b012fe4930116eff0126863ee69e2e92"
Oct 30 07:06:51.609036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.609006    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" event={"ID":"3264b4cb-dde2-45f2-805b-dca581cdfe56","Type":"ContainerStarted","Data":"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"}
Oct 30 07:06:51.609163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.609066    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" event={"ID":"3264b4cb-dde2-45f2-805b-dca581cdfe56","Type":"ContainerStarted","Data":"af324aac0de14fb74c6b352c78b0da4d622841b1bccaf1ef483c687e7d343470"}
Oct 30 07:06:51.609244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.609222    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:06:51.609981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.609949    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:51.610079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.610037    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:51.610143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.610087    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:06:51.618569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.618404    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 07:06:51.811228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.811114    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" podStartSLOduration=1.8110983470000002 podStartE2EDuration="1.811098347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:51.747714929 +0000 UTC m=+572.180478825" watchObservedRunningTime="2026-10-30 07:06:51.811098347 +0000 UTC m=+572.243862240"
Oct 30 07:06:51.811802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.811783    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 07:06:51.814728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.814708    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:51.822508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.822335    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:51.887314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.881761    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-06b26e\""
Oct 30 07:06:51.887314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.881909    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-06b26e-dockercfg-qldd9\""
Oct 30 07:06:51.887314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.884642    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 07:06:51.910574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.910543    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 07:06:51.913921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.913890    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:51.975320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.938778    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 07:06:52.002318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:51.996255    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.007506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.007464    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgtr\" (UniqueName: \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.037416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.037368    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:52.109073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.108969    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.109073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.109019    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.109073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.109074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgtr\" (UniqueName: \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.109397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.109120    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qslr\" (UniqueName: \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.110029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.109991    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.133328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.133270    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgtr\" (UniqueName: \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\") pod \"payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.209754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.209717    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.209966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.209792    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qslr\" (UniqueName: \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.211101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.211069    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.245897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.245855    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qslr\" (UniqueName: \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\") pod \"payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.420079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.420043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:52.430041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.430005    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:52.531408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.531373    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:52.636555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.636417    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" event={"ID":"ca930885-68e5-4124-a913-1550807c0c0b","Type":"ContainerStarted","Data":"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"}
Oct 30 07:06:52.636555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.636471    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" event={"ID":"ca930885-68e5-4124-a913-1550807c0c0b","Type":"ContainerStarted","Data":"c363ada19ee05e3b21cb4fe9d0e03374ee6d5be74a3c9fa46f81040a4df09117"}
Oct 30 07:06:52.638381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.638348    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:06:52.644528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.644484    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerStarted","Data":"cbf2078ef86c10ae5cdfcf32a2e28a4f0199e45154b1d67d497d35b3c539d6ff"}
Oct 30 07:06:52.757246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.756646    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" podStartSLOduration=2.756628957 podStartE2EDuration="2.756628957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:52.685498116 +0000 UTC m=+573.118262010" watchObservedRunningTime="2026-10-30 07:06:52.756628957 +0000 UTC m=+573.189392891"
Oct 30 07:06:52.805317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.802142    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 07:06:52.908795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:52.908665    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 07:06:52.924805 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:06:52.924757    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d426c3b_8ec2_4433_bc09_2a210b93d6b4.slice/crio-11e796a8c43d03dd2ca701cbd00ec0b9a98f1fbf778331aa7500083759ab142b WatchSource:0}: Error finding container 11e796a8c43d03dd2ca701cbd00ec0b9a98f1fbf778331aa7500083759ab142b: Status 404 returned error can't find the container with id 11e796a8c43d03dd2ca701cbd00ec0b9a98f1fbf778331aa7500083759ab142b
Oct 30 07:06:53.567400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.567362    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:53.571486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.571449    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 07:06:53.571762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.571731    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 07:06:53.651596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.651561    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" event={"ID":"7d426c3b-8ec2-4433-bc09-2a210b93d6b4","Type":"ContainerStarted","Data":"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"}
Oct 30 07:06:53.651871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.651851    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" event={"ID":"7d426c3b-8ec2-4433-bc09-2a210b93d6b4","Type":"ContainerStarted","Data":"11e796a8c43d03dd2ca701cbd00ec0b9a98f1fbf778331aa7500083759ab142b"}
Oct 30 07:06:53.652102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.652091    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:06:53.653683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.653657    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" event={"ID":"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33","Type":"ContainerStarted","Data":"b1e55d0c94669eefabb2c2eb72bb1844c343b347c8069bdfcb014e93486f48d8"}
Oct 30 07:06:53.653778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.653707    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" event={"ID":"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33","Type":"ContainerStarted","Data":"b637c154e9a1835b4612cdcbb732abf8f88cb0151bdf809f408cf8b4295756ba"}
Oct 30 07:06:53.654132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.654114    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:06:53.702316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.694566    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" podStartSLOduration=2.694547348 podStartE2EDuration="2.694547348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:53.692828118 +0000 UTC m=+574.125592013" watchObservedRunningTime="2026-10-30 07:06:53.694547348 +0000 UTC m=+574.127311242"
Oct 30 07:06:53.745194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:53.745131    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" podStartSLOduration=2.745116182 podStartE2EDuration="2.745116182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:53.743140679 +0000 UTC m=+574.175904574" watchObservedRunningTime="2026-10-30 07:06:53.745116182 +0000 UTC m=+574.177880074"
Oct 30 07:06:54.519607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:54.519542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:54.581004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:54.580969    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 07:06:54.581651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:54.581618    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 07:06:54.644413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:54.644374    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:54.998177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:54.998143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:55.553274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:55.553242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:55.581253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:55.580779    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 07:06:55.581253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:55.580850    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 07:06:56.588321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:56.571701    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 07:06:56.588321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:56.571782    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 07:06:56.927017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:56.926978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:57.575328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:57.570750    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 07:06:57.575328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:57.570817    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 07:06:57.592929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:57.592890    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 07:06:58.573014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:58.572976    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:58.573355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:58.573327    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:58.574222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:58.574196    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:06:58.750918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:58.750884    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:58.931998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:58.931960    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:59.112148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:59.112105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:59.232204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:59.232105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:59.458694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:59.458664    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:59.586370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:59.586047    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:06:59.708258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:06:59.708206    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:01.119899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.119857    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:01.729181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.729140    6738 generic.go:356] "Generic (PLEG): container finished" podID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerID="3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f" exitCode=0
Oct 30 07:07:01.729382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.729215    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerDied","Data":"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"}
Oct 30 07:07:01.729663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.729645    6738 scope.go:122] "RemoveContainer" containerID="3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"
Oct 30 07:07:01.730994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.730969    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:01.731048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.731036    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:01.731082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:01.731065    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:02.648210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:02.648179    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 07:07:02.737921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:02.737891    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerStarted","Data":"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"}
Oct 30 07:07:03.152637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:03.152599    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:03.725448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:03.725413    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:07:03.725888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:03.725652    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:07:03.725888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:03.725701    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 07:07:04.665283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:04.665250    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 07:07:04.724957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:04.724923    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:07:04.725150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:04.724978    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 07:07:05.187378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:05.187340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:05.698830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:05.698798    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 07:07:05.699007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:05.698854    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 07:07:05.724710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:05.724674    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:07:05.724876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:05.724737    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 07:07:06.094144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:06.094050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:06.725073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:06.725038    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:07:06.725522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:06.725089    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 07:07:07.187276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:07.187239    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:07.331895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:07.331861    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:07.519018 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:07:07.518930    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:07:07.725784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:07.725742    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:07:07.726244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:07.725834    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 07:07:07.998039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:07.997949    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:08.117940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:08.117903    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:08.238351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:08.238316    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:08.729469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:08.729436    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:07:08.729882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:08.729821    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:07:08.730478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:08.730458    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:07:09.354204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:09.354164    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:11.592619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:11.592535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:11.911167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:11.911134    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:12.556598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:12.556561    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:12.694110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:12.694071    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:12.817461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:12.817373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:12.933131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:12.933062    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:13.617253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:13.617154    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:15.439818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:15.439784    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:16.523996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.523859    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:42830->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Oct 30 07:07:16.523996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.523926    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:42830->10.133.0.51:15021: read: connection reset by peer"
Oct 30 07:07:16.877405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.876449    6738 generic.go:356] "Generic (PLEG): container finished" podID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerID="1f936c455a6871cf6b997e8679e50bfa95febda20c6a79445b05a872fb1ba7d7" exitCode=0
Oct 30 07:07:16.877405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.876513    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerDied","Data":"1f936c455a6871cf6b997e8679e50bfa95febda20c6a79445b05a872fb1ba7d7"}
Oct 30 07:07:16.877405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.877080    6738 scope.go:122] "RemoveContainer" containerID="1f936c455a6871cf6b997e8679e50bfa95febda20c6a79445b05a872fb1ba7d7"
Oct 30 07:07:16.878765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.878726    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:16.879003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.878977    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:16.879170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:16.879149    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:07:17.446502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:17.446468    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:17.591495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:17.590284    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:17.896461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:17.896424    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerStarted","Data":"57eb82b5866a8543204cc24e201867ab74eb6d1532a7225c36c3734c1a224163"}
Oct 30 07:07:18.229790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.229704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:18.405062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.405026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:18.443477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.443446    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:07:18.443785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.443654    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:07:18.444078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.444048    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 07:07:18.521515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:18.521428    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:19.433715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:19.433675    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:07:19.434226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:19.433732    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 07:07:19.547333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:19.547264    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:20.432851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:20.432814    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:07:20.433046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:20.432893    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 07:07:21.432803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:21.432754    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:07:21.433381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:21.432828    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 07:07:21.448988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:21.448951    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:22.433218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:22.433180    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:07:22.433653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:22.433236    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 07:07:23.354732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:23.354690    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:23.437315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:23.437271    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:07:23.437770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:23.437758    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:07:23.438563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:23.438531    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:07:24.309267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.309191    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 07:07:24.314089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.314059    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.334739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.334706    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 07:07:24.344922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.344891    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"openshift-service-ca.crt\""
Oct 30 07:07:24.345078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.344925    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"kube-root-ca.crt\""
Oct 30 07:07:24.345078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.345021    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"default-dockercfg-7chhx\""
Oct 30 07:07:24.345620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.345605    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"simulator-w1-4fe2ed-kserve-self-signed-certs\""
Oct 30 07:07:24.380952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.380921    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.380972    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.381157    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkt4\" (UniqueName: \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.381218    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.381253    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.381282    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.381818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.381787    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:24.482097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482058    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482196    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkt4\" (UniqueName: \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482262    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482309    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482434    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482540    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482659    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482767    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.482933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.482839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.484948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.484927    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.485026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.484932    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.506659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.506622    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkt4\" (UniqueName: \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\") pod \"simulator-w1-4fe2ed-kserve-55f568947b-gd4mf\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") " pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.627895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.627862    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:24.931726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:24.931646    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 07:07:24.935432 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:07:24.935402    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0b4fd7e_470b_424c_8a3f_79f7b2d302ed.slice/crio-689dece6e6ee41cb839bc15f319a7c8dc84495f8d60530c79b34153da34bf396 WatchSource:0}: Error finding container 689dece6e6ee41cb839bc15f319a7c8dc84495f8d60530c79b34153da34bf396: Status 404 returned error can't find the container with id 689dece6e6ee41cb839bc15f319a7c8dc84495f8d60530c79b34153da34bf396
Oct 30 07:07:25.233457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:25.233359    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:25.942822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:25.942783    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" event={"ID":"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed","Type":"ContainerStarted","Data":"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"}
Oct 30 07:07:25.943348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:25.942831    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" event={"ID":"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed","Type":"ContainerStarted","Data":"689dece6e6ee41cb839bc15f319a7c8dc84495f8d60530c79b34153da34bf396"}
Oct 30 07:07:25.986640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:25.986581    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podStartSLOduration=1.986564967 podStartE2EDuration="1.986564967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:25.985217278 +0000 UTC m=+606.417981170" watchObservedRunningTime="2026-10-30 07:07:25.986564967 +0000 UTC m=+606.419328859"
Oct 30 07:07:26.628208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.628159    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:26.799872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.799831    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 07:07:26.803430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.803406    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.833429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.833375    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"default-dockercfg-x9pcs\""
Oct 30 07:07:26.833429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.833398    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"kube-root-ca.crt\""
Oct 30 07:07:26.833429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.833405    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"openshift-service-ca.crt\""
Oct 30 07:07:26.833746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.833438    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"premium-w3-2eda2c-kserve-self-signed-certs\""
Oct 30 07:07:26.835284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.835250    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 07:07:26.872223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.872187    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:26.921581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921500    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.921581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921534    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.921762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921678    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.921762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921747    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.921832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921794    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:26.921866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:26.921833    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94blp\" (UniqueName: \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.021213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.021169    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:27.023376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023351    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023412    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023429    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023451    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94blp\" (UniqueName: \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023503    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023531    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023785    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023834    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.023910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.023889    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.025822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.025799    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.025956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.025941    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.047540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.047504    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94blp\" (UniqueName: \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\") pod \"premium-w3-2eda2c-kserve-7f964dbd4-4djcv\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") " pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.116863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.116816    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:27.417616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.417582    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 07:07:27.419139 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:07:27.419107    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fb29921_bfb1_4e15_b66f_ebb3c6dd05fb.slice/crio-4ffc885bb6de5a55273e594959495557cdfb71cf4f27fd34e123dceea55d7479 WatchSource:0}: Error finding container 4ffc885bb6de5a55273e594959495557cdfb71cf4f27fd34e123dceea55d7479: Status 404 returned error can't find the container with id 4ffc885bb6de5a55273e594959495557cdfb71cf4f27fd34e123dceea55d7479
Oct 30 07:07:27.956434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.956390    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" event={"ID":"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb","Type":"ContainerStarted","Data":"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"}
Oct 30 07:07:27.956630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:27.956443    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" event={"ID":"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb","Type":"ContainerStarted","Data":"4ffc885bb6de5a55273e594959495557cdfb71cf4f27fd34e123dceea55d7479"}
Oct 30 07:07:28.000372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:28.000305    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podStartSLOduration=2.000276512 podStartE2EDuration="2.000276512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:27.998229736 +0000 UTC m=+608.430993628" watchObservedRunningTime="2026-10-30 07:07:28.000276512 +0000 UTC m=+608.433040404"
Oct 30 07:07:28.107783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:28.107745    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:28.117131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:28.117089    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:29.039819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:29.039787    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:29.829868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:29.829836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:30.941630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:30.941593    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:32.830688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:32.830653    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:34.591954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:34.591917    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:34.639961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:34.639924    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:34.662150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:34.662120    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:35.007732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:35.007702    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:07:36.453581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:36.453535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:37.097284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:37.097247    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:37.117417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:37.117375    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:37.130529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:37.130496    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:37.517151 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:07:37.517112    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:07:38.023328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:38.023276    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:07:38.410049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:38.410011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:40.236641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:40.236606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:42.143225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:42.143188    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:44.050095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:44.050058    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:45.945105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:45.945057    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:46.725596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:46.725557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:47.845618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:47.845580    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:49.784572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:49.784533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:50.675676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:50.675617    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:51.642201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:51.642162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:53.486459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:53.486420    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:55.334102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:55.334044    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:57.422524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:57.422480    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:07:59.250640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:07:59.250547    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:01.035608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:01.035569    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:02.555349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.552116    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 07:08:02.555900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.555878    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.584967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.584927    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"premium-w1-4fe2ed-kserve-self-signed-certs\""
Oct 30 07:08:02.586817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.586789    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 07:08:02.615310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.615256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:02.670188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670148    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.670405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670215    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.670405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670326    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcc9\" (UniqueName: \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.670405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670377    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.670405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670402    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.670605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.670418    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772529    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772689    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772724    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcc9\" (UniqueName: \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772776    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772820    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.772845    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.775341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.773543    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.777579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.777550    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.778567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.778532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.778793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.778768    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.782699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.782657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.802965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.802934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcc9\" (UniqueName: \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\") pod \"premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") " pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:02.854661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.854550    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:02.870658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:02.870628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:03.196032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:03.196008    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 07:08:03.198308 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:03.198257    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff76749_6812_40a4_aa07_9d3ecb6ba0ff.slice/crio-746ba0ed0e3dc0b7b127b3a598d4b985e2d64bc49cda19e2980eb5f33c1c4733 WatchSource:0}: Error finding container 746ba0ed0e3dc0b7b127b3a598d4b985e2d64bc49cda19e2980eb5f33c1c4733: Status 404 returned error can't find the container with id 746ba0ed0e3dc0b7b127b3a598d4b985e2d64bc49cda19e2980eb5f33c1c4733
Oct 30 07:08:04.154620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.154579    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" event={"ID":"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff","Type":"ContainerStarted","Data":"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"}
Oct 30 07:08:04.155018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.154631    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" event={"ID":"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff","Type":"ContainerStarted","Data":"746ba0ed0e3dc0b7b127b3a598d4b985e2d64bc49cda19e2980eb5f33c1c4733"}
Oct 30 07:08:04.198551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.198498    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podStartSLOduration=2.198482876 podStartE2EDuration="2.198482876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:04.196909886 +0000 UTC m=+644.629673779" watchObservedRunningTime="2026-10-30 07:08:04.198482876 +0000 UTC m=+644.631246768"
Oct 30 07:08:04.802219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.802189    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:04.871097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.871058    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:04.932021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:04.931980    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:06.877216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:06.877176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:07.517353 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:08:07.517316    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:08:08.747728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:08.747691    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:10.741990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:10.741952    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:12.637808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:12.637767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:12.870993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:12.870960    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:12.884096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:12.884067    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:13.223241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:13.223211    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:08:14.534271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:14.534229    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:16.324553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:16.324515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:18.536263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:18.536223    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:20.572048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:20.570612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:22.467519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:22.467484    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:24.345865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:24.345824    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:25.590344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.590306    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"]
Oct 30 07:08:25.593523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.593499    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.612822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.612783    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:08:25.612979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.612863    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:08:25.613162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.613145    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:08:25.619734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.619700    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.619891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.619770    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tmb\" (UniqueName: \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.619891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.619872    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.624937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.624904    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:08:25.626805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.626770    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"]
Oct 30 07:08:25.721202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.721166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tmb\" (UniqueName: \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.721202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.721200    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.721494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.721399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.722112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.722084    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.723528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.723510    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.745367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.745342    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tmb\" (UniqueName: \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\") pod \"maas-api-revoke-sub-beb7fc9e-vbgjf\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:25.906444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:25.906412    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:26.206763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:26.206732    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"]
Oct 30 07:08:26.209385 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:26.209356    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162da9e3_dc50_4df5_b4b6_afdd779d5708.slice/crio-bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a WatchSource:0}: Error finding container bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a: Status 404 returned error can't find the container with id bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a
Oct 30 07:08:26.283484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:26.283447    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf" event={"ID":"162da9e3-dc50-4df5-b4b6-afdd779d5708","Type":"ContainerStarted","Data":"bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a"}
Oct 30 07:08:26.358745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:26.358713    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:27.290323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:27.290269    6738 generic.go:356] "Generic (PLEG): container finished" podID="162da9e3-dc50-4df5-b4b6-afdd779d5708" containerID="a42a8f7d5391230ef5a22e732c54355955bf884b2e5c2e7ba65ed6505b68c84e" exitCode=0
Oct 30 07:08:27.290719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:27.290351    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf" event={"ID":"162da9e3-dc50-4df5-b4b6-afdd779d5708","Type":"ContainerDied","Data":"a42a8f7d5391230ef5a22e732c54355955bf884b2e5c2e7ba65ed6505b68c84e"}
Oct 30 07:08:28.156367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.156333    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:28.282489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.282452    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:28.446880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.446853    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:28.549806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.549773    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\") pod \"162da9e3-dc50-4df5-b4b6-afdd779d5708\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") "
Oct 30 07:08:28.550003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.549827    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\" (UniqueName: \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\") pod \"162da9e3-dc50-4df5-b4b6-afdd779d5708\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") "
Oct 30 07:08:28.550003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.549983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\") pod \"162da9e3-dc50-4df5-b4b6-afdd779d5708\" (UID: \"162da9e3-dc50-4df5-b4b6-afdd779d5708\") "
Oct 30 07:08:28.550365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.550338    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca" pod "162da9e3-dc50-4df5-b4b6-afdd779d5708" (UID: "162da9e3-dc50-4df5-b4b6-afdd779d5708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:08:28.550488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.550453    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:28.551959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.551934    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca" pod "162da9e3-dc50-4df5-b4b6-afdd779d5708" (UID: "162da9e3-dc50-4df5-b4b6-afdd779d5708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:08:28.552060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.552020    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb" pod "162da9e3-dc50-4df5-b4b6-afdd779d5708" (UID: "162da9e3-dc50-4df5-b4b6-afdd779d5708"). InnerVolumeSpecName "kube-api-access-78tmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:08:28.651975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.651931    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162da9e3-dc50-4df5-b4b6-afdd779d5708-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:28.651975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:28.651967    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tmb\" (UniqueName: \"kubernetes.io/projected/162da9e3-dc50-4df5-b4b6-afdd779d5708-kube-api-access-78tmb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:29.302729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:29.302696    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"
Oct 30 07:08:29.302729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:29.302722    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf" event={"ID":"162da9e3-dc50-4df5-b4b6-afdd779d5708","Type":"ContainerDied","Data":"bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a"}
Oct 30 07:08:29.302974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:29.302791    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1e1c5f2423c359e9afeea72899d5d5559edc40f80db4817084e3fa820fe97a"
Oct 30 07:08:30.044834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:30.044782    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:32.084034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:32.083984    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:34.253501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:34.253449    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:36.156672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:36.156634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:37.517924 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:08:37.517886    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:08:38.070531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:38.070495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:39.848985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:39.848948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:40.740492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.740456    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"]
Oct 30 07:08:40.740949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.740934    6738 cpu_manager.go:414] "containerMap: removing container" podUID="162da9e3-dc50-4df5-b4b6-afdd779d5708" containerName="revoke-keys"
Oct 30 07:08:40.741013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.740952    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162da9e3-dc50-4df5-b4b6-afdd779d5708" containerName="revoke-keys"
Oct 30 07:08:40.741066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.741050    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="162da9e3-dc50-4df5-b4b6-afdd779d5708" containerName="revoke-keys"
Oct 30 07:08:40.743317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.743285    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.760451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.760394    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:08:40.772259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.772230    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:08:40.772454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.772258    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:08:40.772454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.772337    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:08:40.774165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.774144    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"]
Oct 30 07:08:40.885497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.885463    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.885885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.885597    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llnm\" (UniqueName: \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.885885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.885639    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.987025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.986980    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.987215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.987106    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llnm\" (UniqueName: \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.987215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.987145    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.987755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.987733    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:40.989644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:40.989623    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:41.014459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:41.014381    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llnm\" (UniqueName: \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\") pod \"maas-api-revoke-sub-37907a09-ctgrh\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:41.056376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:41.056340    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:41.357384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:41.357354    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"]
Oct 30 07:08:41.359202 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:41.359173    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d989e68_b813_43f7_934c_01fdb8a8bfed.slice/crio-833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14 WatchSource:0}: Error finding container 833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14: Status 404 returned error can't find the container with id 833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14
Oct 30 07:08:41.373645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:41.373608    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh" event={"ID":"8d989e68-b813-43f7-934c-01fdb8a8bfed","Type":"ContainerStarted","Data":"833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14"}
Oct 30 07:08:41.728690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:41.728654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:42.383621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:42.383577    6738 generic.go:356] "Generic (PLEG): container finished" podID="8d989e68-b813-43f7-934c-01fdb8a8bfed" containerID="ac9fc93e1cba162ab7dd5979c37a3b07a7edbdd6ab001e99a366bff96b673b8a" exitCode=0
Oct 30 07:08:42.384161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:42.383660    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh" event={"ID":"8d989e68-b813-43f7-934c-01fdb8a8bfed","Type":"ContainerDied","Data":"ac9fc93e1cba162ab7dd5979c37a3b07a7edbdd6ab001e99a366bff96b673b8a"}
Oct 30 07:08:43.253371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.253330    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:08:43.256980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.256931    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.273586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.273560    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe2ed-openshift-default-dockercfg-s6hbq\""
Oct 30 07:08:43.296659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.289526    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:08:43.313354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313281    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313364    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313482    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313522    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313633    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313674    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313702    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313808    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.313878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.313839    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5f4\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415453    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415500    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5f4\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415583    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415700    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415755    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415784    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415832    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415864    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.415946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.415897    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.416375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.416043    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.416375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.416217    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.416375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.416266    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.416375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.416361    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.416521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.416416    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.418647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.418620    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.418785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.418736    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.439402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.439373    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.439779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.439757    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5f4\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.514614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.514585    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:43.569840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.569810    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:43.609252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.609214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:43.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.617955    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\") pod \"8d989e68-b813-43f7-934c-01fdb8a8bfed\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") "
Oct 30 07:08:43.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.618016    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\" (UniqueName: \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\") pod \"8d989e68-b813-43f7-934c-01fdb8a8bfed\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") "
Oct 30 07:08:43.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.618117    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\") pod \"8d989e68-b813-43f7-934c-01fdb8a8bfed\" (UID: \"8d989e68-b813-43f7-934c-01fdb8a8bfed\") "
Oct 30 07:08:43.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.618460    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca" pod "8d989e68-b813-43f7-934c-01fdb8a8bfed" (UID: "8d989e68-b813-43f7-934c-01fdb8a8bfed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:08:43.620795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.620769    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm" pod "8d989e68-b813-43f7-934c-01fdb8a8bfed" (UID: "8d989e68-b813-43f7-934c-01fdb8a8bfed"). InnerVolumeSpecName "kube-api-access-4llnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:08:43.621663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.621640    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca" pod "8d989e68-b813-43f7-934c-01fdb8a8bfed" (UID: "8d989e68-b813-43f7-934c-01fdb8a8bfed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:08:43.719979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.719932    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:43.719979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.719968    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d989e68-b813-43f7-934c-01fdb8a8bfed-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:43.719979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.719981    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4llnm\" (UniqueName: \"kubernetes.io/projected/8d989e68-b813-43f7-934c-01fdb8a8bfed-kube-api-access-4llnm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:43.890857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.890825    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:08:43.892253 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:43.892224    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433c89c0_251b_4a82_8ea2_125f8c3ecd05.slice/crio-6047395e209d15cde52e270529fef8a44d6282de14678f2cef27367e69992658 WatchSource:0}: Error finding container 6047395e209d15cde52e270529fef8a44d6282de14678f2cef27367e69992658: Status 404 returned error can't find the container with id 6047395e209d15cde52e270529fef8a44d6282de14678f2cef27367e69992658
Oct 30 07:08:43.894660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.894626    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:43.894730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.894694    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:43.894730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:43.894721    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:44.397513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.397475    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" event={"ID":"433c89c0-251b-4a82-8ea2-125f8c3ecd05","Type":"ContainerStarted","Data":"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"}
Oct 30 07:08:44.397728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.397533    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" event={"ID":"433c89c0-251b-4a82-8ea2-125f8c3ecd05","Type":"ContainerStarted","Data":"6047395e209d15cde52e270529fef8a44d6282de14678f2cef27367e69992658"}
Oct 30 07:08:44.398890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.398864    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh" event={"ID":"8d989e68-b813-43f7-934c-01fdb8a8bfed","Type":"ContainerDied","Data":"833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14"}
Oct 30 07:08:44.398890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.398883    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"
Oct 30 07:08:44.399036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.398910    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833d878fc1cf0f2daa03f8241fbf498ece65333afc125c9b15a9e97b4f01ee14"
Oct 30 07:08:44.446398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.446320    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podStartSLOduration=1.446285963 podStartE2EDuration="1.446285963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:44.444635725 +0000 UTC m=+684.877399622" watchObservedRunningTime="2026-10-30 07:08:44.446285963 +0000 UTC m=+684.879049859"
Oct 30 07:08:44.584317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.572357    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:44.584317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.581447    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:44.584317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:44.581513    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:45.579319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:45.570730    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:45.579319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:45.570796    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:45.698858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:45.698416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:46.580318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:46.575752    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:46.580318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:46.575868    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:47.587567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:47.578724    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:47.587567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:47.578784    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:47.693813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:47.693758    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:48.577320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:48.575737    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:48.577320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:48.575804    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:48.775790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:48.775752    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:49.593352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:49.580678    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:08:49.593352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:49.580740    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 07:08:49.679325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:49.677115    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:50.065889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.065853    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:08:50.066372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.066356    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8d989e68-b813-43f7-934c-01fdb8a8bfed" containerName="revoke-keys"
Oct 30 07:08:50.066448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.066373    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d989e68-b813-43f7-934c-01fdb8a8bfed" containerName="revoke-keys"
Oct 30 07:08:50.066484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.066468    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d989e68-b813-43f7-934c-01fdb8a8bfed" containerName="revoke-keys"
Oct 30 07:08:50.068983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.068965    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.087109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.087088    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb6be4-openshift-default-dockercfg-2kstl\""
Oct 30 07:08:50.098818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.098784    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:08:50.208109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208060    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208171    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208205    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208236    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzw52\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208381    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208431    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208544    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208640    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.208690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.208676    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.309957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.309919    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310001    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310034    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310079    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310122    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310159    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310191    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzw52\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310225    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310267    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310472    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.310766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.310737    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.311215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.311192    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.311397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.311241    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.311560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.311540    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.312928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.312904    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.313079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.313062    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.336033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.335931    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.336033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.335931    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzw52\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.381820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.381780    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:50.599582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.599481    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:50.599919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.599846    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:50.601191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.601163    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:08:50.606823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.606793    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"]
Oct 30 07:08:50.610521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.610497    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.626468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.626436    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:08:50.626704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.626677    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:08:50.626881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.626861    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:08:50.627064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.626964    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:08:50.629449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.629426    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"]
Oct 30 07:08:50.684703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.684666    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"]
Oct 30 07:08:50.690543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.690514    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.707092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.707066    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:08:50.708082 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:50.708053    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ea3421_c4fe_44cd_80e4_b9e19507209f.slice/crio-a42feec5e84ceae32516ac1978c8e163ca18dead9d358b8c7864a518d3799eda WatchSource:0}: Error finding container a42feec5e84ceae32516ac1978c8e163ca18dead9d358b8c7864a518d3799eda: Status 404 returned error can't find the container with id a42feec5e84ceae32516ac1978c8e163ca18dead9d358b8c7864a518d3799eda
Oct 30 07:08:50.709911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.709884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"]
Oct 30 07:08:50.712126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.712081    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:50.712226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.712163    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:50.712226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.712203    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:08:50.715170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.715144    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.715430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.715404    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.715544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.715451    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5vc\" (UniqueName: \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.816282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816228    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhqj\" (UniqueName: \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.816482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816368    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.816553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816482    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.816608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816561    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.816608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816590    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.816711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.816627    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5vc\" (UniqueName: \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.817816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.817787    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.819723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.819690    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.840764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.840698    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5vc\" (UniqueName: \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\") pod \"maas-api-revoke-sub-c9900ca1-rvx6s\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.917902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.917861    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.918093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.917922    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhqj\" (UniqueName: \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.918093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.917966    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.918696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.918671    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.920726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.920705    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:50.925748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.925715    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:50.945772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:50.945735    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhqj\" (UniqueName: \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\") pod \"maas-api-revoke-sub-cc002b8a-lbjnh\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:51.015388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.015107    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:51.232954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.232925    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"]
Oct 30 07:08:51.234408 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:51.234376    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1814cc7d_3186_4c87_bc24_1646213dc06b.slice/crio-ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c WatchSource:0}: Error finding container ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c: Status 404 returned error can't find the container with id ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c
Oct 30 07:08:51.327585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.327546    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"]
Oct 30 07:08:51.328227 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:51.328193    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d70618_8d1a_4ae6_a5e0_ee893ae9e3b9.slice/crio-ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86 WatchSource:0}: Error finding container ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86: Status 404 returned error can't find the container with id ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86
Oct 30 07:08:51.441251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.441205    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" event={"ID":"f3ea3421-c4fe-44cd-80e4-b9e19507209f","Type":"ContainerStarted","Data":"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"}
Oct 30 07:08:51.441481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.441271    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" event={"ID":"f3ea3421-c4fe-44cd-80e4-b9e19507209f","Type":"ContainerStarted","Data":"a42feec5e84ceae32516ac1978c8e163ca18dead9d358b8c7864a518d3799eda"}
Oct 30 07:08:51.442963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.442931    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s" event={"ID":"1814cc7d-3186-4c87-bc24-1646213dc06b","Type":"ContainerStarted","Data":"d1877e2b8db6937c7084e2d4d5a737811db03a8cd2c10a6e529a1e7a06337c19"}
Oct 30 07:08:51.443095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.442975    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s" event={"ID":"1814cc7d-3186-4c87-bc24-1646213dc06b","Type":"ContainerStarted","Data":"ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c"}
Oct 30 07:08:51.444261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.444237    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh" event={"ID":"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9","Type":"ContainerStarted","Data":"ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86"}
Oct 30 07:08:51.489842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.489711    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podStartSLOduration=1.489692201 podStartE2EDuration="1.489692201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:51.486585113 +0000 UTC m=+691.919349009" watchObservedRunningTime="2026-10-30 07:08:51.489692201 +0000 UTC m=+691.922456170"
Oct 30 07:08:51.533340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.529628    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s" podStartSLOduration=1.529610954 podStartE2EDuration="1.529610954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:51.52786916 +0000 UTC m=+691.960633054" watchObservedRunningTime="2026-10-30 07:08:51.529610954 +0000 UTC m=+691.962374847"
Oct 30 07:08:51.643898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:51.643864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:52.393725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.393595    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:52.395649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.395615    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:52.395825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.395676    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:52.452495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.452458    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh" event={"ID":"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9","Type":"ContainerStarted","Data":"54b79d8e5e79fa6483e87052841937834e8567e16b8fbab70f2dcb6ba6317525"}
Oct 30 07:08:52.454187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.454158    6738 generic.go:356] "Generic (PLEG): container finished" podID="1814cc7d-3186-4c87-bc24-1646213dc06b" containerID="d1877e2b8db6937c7084e2d4d5a737811db03a8cd2c10a6e529a1e7a06337c19" exitCode=0
Oct 30 07:08:52.454380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.454234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s" event={"ID":"1814cc7d-3186-4c87-bc24-1646213dc06b","Type":"ContainerDied","Data":"d1877e2b8db6937c7084e2d4d5a737811db03a8cd2c10a6e529a1e7a06337c19"}
Oct 30 07:08:52.519981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.519925    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh" podStartSLOduration=2.519903599 podStartE2EDuration="2.519903599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:52.519043426 +0000 UTC m=+692.951807319" watchObservedRunningTime="2026-10-30 07:08:52.519903599 +0000 UTC m=+692.952667492"
Oct 30 07:08:52.655910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.655822    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"]
Oct 30 07:08:52.658768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.658747    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.700930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.700884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"]
Oct 30 07:08:52.747050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.747009    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.747232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.747138    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnbp\" (UniqueName: \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.747232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.747200    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.848591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.848550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.848790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.848733    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnbp\" (UniqueName: \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.848861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.848841    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.849538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.849515    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.851043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.851019    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.894785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.894755    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnbp\" (UniqueName: \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\") pod \"maas-api-revoke-sub-6e9f912e-9fzkr\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:52.996767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:52.996686    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:53.300562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.300526    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"]
Oct 30 07:08:53.302964 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:08:53.302927    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a40e5c_7647_4525_9cf9_276a20f00509.slice/crio-2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd WatchSource:0}: Error finding container 2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd: Status 404 returned error can't find the container with id 2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd
Oct 30 07:08:53.386318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.384754    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:53.386318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.384828    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:53.461620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.461582    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr" event={"ID":"01a40e5c-7647-4525-9cf9-276a20f00509","Type":"ContainerStarted","Data":"2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd"}
Oct 30 07:08:53.463203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.463178    6738 generic.go:356] "Generic (PLEG): container finished" podID="39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" containerID="54b79d8e5e79fa6483e87052841937834e8567e16b8fbab70f2dcb6ba6317525" exitCode=0
Oct 30 07:08:53.463339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.463267    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh" event={"ID":"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9","Type":"ContainerDied","Data":"54b79d8e5e79fa6483e87052841937834e8567e16b8fbab70f2dcb6ba6317525"}
Oct 30 07:08:53.776350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.775539    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:53.807379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.801176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:53.878996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.878946    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\") pod \"1814cc7d-3186-4c87-bc24-1646213dc06b\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") "
Oct 30 07:08:53.879342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.879318    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\") pod \"1814cc7d-3186-4c87-bc24-1646213dc06b\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") "
Oct 30 07:08:53.879641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.879624    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\" (UniqueName: \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\") pod \"1814cc7d-3186-4c87-bc24-1646213dc06b\" (UID: \"1814cc7d-3186-4c87-bc24-1646213dc06b\") "
Oct 30 07:08:53.927902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.927853    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca" pod "1814cc7d-3186-4c87-bc24-1646213dc06b" (UID: "1814cc7d-3186-4c87-bc24-1646213dc06b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:08:53.934317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.929742    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca" pod "1814cc7d-3186-4c87-bc24-1646213dc06b" (UID: "1814cc7d-3186-4c87-bc24-1646213dc06b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:08:53.938692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.938632    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc" pod "1814cc7d-3186-4c87-bc24-1646213dc06b" (UID: "1814cc7d-3186-4c87-bc24-1646213dc06b"). InnerVolumeSpecName "kube-api-access-9l5vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:08:53.998151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.998058    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:53.998151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.998096    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1814cc7d-3186-4c87-bc24-1646213dc06b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:53.998151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:53.998110    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l5vc\" (UniqueName: \"kubernetes.io/projected/1814cc7d-3186-4c87-bc24-1646213dc06b-kube-api-access-9l5vc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:54.396319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.385347    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:54.396319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.385423    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:54.520570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.520125    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr" event={"ID":"01a40e5c-7647-4525-9cf9-276a20f00509","Type":"ContainerStarted","Data":"5a7558a06b1d954a810ed0e06876edf3c61c152423dc64916eac5c9f1457534a"}
Oct 30 07:08:54.542353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.539649    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"
Oct 30 07:08:54.543319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.543260    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s" event={"ID":"1814cc7d-3186-4c87-bc24-1646213dc06b","Type":"ContainerDied","Data":"ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c"}
Oct 30 07:08:54.543444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.543333    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddafcda0ece79d6dfde1675de8c0ed4e5d2c58f48241e0abeecdf0172a02e56c"
Oct 30 07:08:54.741772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.741742    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:54.795409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.795347    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr" podStartSLOduration=2.795326298 podStartE2EDuration="2.795326298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:54.575630234 +0000 UTC m=+695.008394129" watchObservedRunningTime="2026-10-30 07:08:54.795326298 +0000 UTC m=+695.228090193"
Oct 30 07:08:54.920822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.920772    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\") pod \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") "
Oct 30 07:08:54.920998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.920925    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\") pod \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") "
Oct 30 07:08:54.920998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.920966    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\" (UniqueName: \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\") pod \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\" (UID: \"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9\") "
Oct 30 07:08:54.921493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.921467    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca" pod "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" (UID: "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:08:54.922951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.922925    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca" pod "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" (UID: "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:08:54.923147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:54.923125    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj" pod "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" (UID: "39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9"). InnerVolumeSpecName "kube-api-access-hkhqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:08:55.022497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.022399    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:55.022497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.022436    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:55.022497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.022451    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhqj\" (UniqueName: \"kubernetes.io/projected/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9-kube-api-access-hkhqj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:55.391329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.389764    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:55.391329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.389827    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:55.546376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.546349    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"
Oct 30 07:08:55.546825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.546368    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh" event={"ID":"39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9","Type":"ContainerDied","Data":"ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86"}
Oct 30 07:08:55.546825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.546423    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea40c02ff466edd41596db6d4a43c503288de9756b4cc9f36b7a8f6991f66d86"
Oct 30 07:08:55.547709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.547682    6738 generic.go:356] "Generic (PLEG): container finished" podID="01a40e5c-7647-4525-9cf9-276a20f00509" containerID="5a7558a06b1d954a810ed0e06876edf3c61c152423dc64916eac5c9f1457534a" exitCode=0
Oct 30 07:08:55.547903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.547885    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr" event={"ID":"01a40e5c-7647-4525-9cf9-276a20f00509","Type":"ContainerDied","Data":"5a7558a06b1d954a810ed0e06876edf3c61c152423dc64916eac5c9f1457534a"}
Oct 30 07:08:55.901090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:55.901050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:56.389380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.385706    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:56.389380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.385764    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:56.728879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.728850    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:56.845247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.845205    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\") pod \"01a40e5c-7647-4525-9cf9-276a20f00509\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") "
Oct 30 07:08:56.845439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.845308    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\" (UniqueName: \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\") pod \"01a40e5c-7647-4525-9cf9-276a20f00509\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") "
Oct 30 07:08:56.845439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.845358    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\") pod \"01a40e5c-7647-4525-9cf9-276a20f00509\" (UID: \"01a40e5c-7647-4525-9cf9-276a20f00509\") "
Oct 30 07:08:56.846699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.846671    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca" pod "01a40e5c-7647-4525-9cf9-276a20f00509" (UID: "01a40e5c-7647-4525-9cf9-276a20f00509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:08:56.847577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.847556    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca" pod "01a40e5c-7647-4525-9cf9-276a20f00509" (UID: "01a40e5c-7647-4525-9cf9-276a20f00509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:08:56.847684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.847671    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp" pod "01a40e5c-7647-4525-9cf9-276a20f00509" (UID: "01a40e5c-7647-4525-9cf9-276a20f00509"). InnerVolumeSpecName "kube-api-access-jcnbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:08:56.946432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.946397    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:56.946432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.946427    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnbp\" (UniqueName: \"kubernetes.io/projected/01a40e5c-7647-4525-9cf9-276a20f00509-kube-api-access-jcnbp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:56.946432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:56.946439    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a40e5c-7647-4525-9cf9-276a20f00509-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:08:57.382724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.382689    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:08:57.382941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.382745    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 07:08:57.564537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.564509    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"
Oct 30 07:08:57.564709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.564542    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr" event={"ID":"01a40e5c-7647-4525-9cf9-276a20f00509","Type":"ContainerDied","Data":"2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd"}
Oct 30 07:08:57.564709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.564609    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd8651ac1af70273caf778ef09b0a02e5b83d45f58511691ac75c3130310fcd"
Oct 30 07:08:57.902285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:57.902248    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:08:58.495966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:58.495928    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:58.496415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:58.496391    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:58.497662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:58.497642    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:08:59.843419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:08:59.843375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:01.847091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:01.847046    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:03.771482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:03.771441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:05.323854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.323812    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"]
Oct 30 07:09:05.324366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324340    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1814cc7d-3186-4c87-bc24-1646213dc06b" containerName="revoke-keys"
Oct 30 07:09:05.324366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324353    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1814cc7d-3186-4c87-bc24-1646213dc06b" containerName="revoke-keys"
Oct 30 07:09:05.324366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324362    6738 cpu_manager.go:414] "containerMap: removing container" podUID="39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" containerName="revoke-keys"
Oct 30 07:09:05.324366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324368    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" containerName="revoke-keys"
Oct 30 07:09:05.324588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324384    6738 cpu_manager.go:414] "containerMap: removing container" podUID="01a40e5c-7647-4525-9cf9-276a20f00509" containerName="revoke-keys"
Oct 30 07:09:05.324588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324390    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a40e5c-7647-4525-9cf9-276a20f00509" containerName="revoke-keys"
Oct 30 07:09:05.324588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324455    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a40e5c-7647-4525-9cf9-276a20f00509" containerName="revoke-keys"
Oct 30 07:09:05.324588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324468    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1814cc7d-3186-4c87-bc24-1646213dc06b" containerName="revoke-keys"
Oct 30 07:09:05.324588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.324475    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" containerName="revoke-keys"
Oct 30 07:09:05.327055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.327032    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.345925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.345893    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"]
Oct 30 07:09:05.355926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.355900    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:09:05.356082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.355900    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:09:05.356082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.355916    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:09:05.356082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.355986    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:09:05.359696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.359670    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkjm\" (UniqueName: \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.359822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.359705    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.359822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.359812    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.460650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.460613    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.460857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.460738    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkjm\" (UniqueName: \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.460857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.460765    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.461353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.461333    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.463069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.463045    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.488501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.488465    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkjm\" (UniqueName: \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\") pod \"maas-api-revoke-sub-37061ad2-nvn2v\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.493869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.493822    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:09:05.497439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.497416    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.513972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.513946    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"e2e-conflict-a-2e055bbe-kserve-self-signed-certs\""
Oct 30 07:09:05.516915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.516884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:09:05.561452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561416    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.561452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561455    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.561683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561475    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.561683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561594    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.561683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561644    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.561897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.561876    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5b5\" (UniqueName: \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.562636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.562612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:05.639147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.639119    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:05.662609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662574    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5b5\" (UniqueName: \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.662772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662655    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.662772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662684    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.662772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662700    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.662772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662717    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.662997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.662827    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.663123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.663101    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.663194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.663123    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.663244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.663217    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.665287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.665260    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.665835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.665812    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.673268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.673232    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:05.686977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.686947    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5b5\" (UniqueName: \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\") pod \"e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.819580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.819540    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:05.943143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:05.943113    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"]
Oct 30 07:09:05.946417 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:09:05.946388    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b968b3_3fa2_4309_b085_ca6a155b363c.slice/crio-c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23 WatchSource:0}: Error finding container c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23: Status 404 returned error can't find the container with id c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23
Oct 30 07:09:06.130468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.130441    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:09:06.131265 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:09:06.131237    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54990866_533d_4911_8fb2_cadc0abb2e4f.slice/crio-50ff8134a86374bc082f299a85041e58416d30eb9f3f98decf71e7c013e92b92 WatchSource:0}: Error finding container 50ff8134a86374bc082f299a85041e58416d30eb9f3f98decf71e7c013e92b92: Status 404 returned error can't find the container with id 50ff8134a86374bc082f299a85041e58416d30eb9f3f98decf71e7c013e92b92
Oct 30 07:09:06.649276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.649234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v" event={"ID":"24b968b3-3fa2-4309-b085-ca6a155b363c","Type":"ContainerStarted","Data":"28bb002af912e63da1d27f28b8c6036947f8f9702e1490087c2c9e3e2b9a33d2"}
Oct 30 07:09:06.649724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.649344    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v" event={"ID":"24b968b3-3fa2-4309-b085-ca6a155b363c","Type":"ContainerStarted","Data":"c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23"}
Oct 30 07:09:06.650827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.650799    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" event={"ID":"54990866-533d-4911-8fb2-cadc0abb2e4f","Type":"ContainerStarted","Data":"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"}
Oct 30 07:09:06.650952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.650837    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" event={"ID":"54990866-533d-4911-8fb2-cadc0abb2e4f","Type":"ContainerStarted","Data":"50ff8134a86374bc082f299a85041e58416d30eb9f3f98decf71e7c013e92b92"}
Oct 30 07:09:06.690627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.690575    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v" podStartSLOduration=1.6905613069999998 podStartE2EDuration="1.690561307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:06.690316995 +0000 UTC m=+707.123080880" watchObservedRunningTime="2026-10-30 07:09:06.690561307 +0000 UTC m=+707.123325200"
Oct 30 07:09:06.734562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.734511    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" podStartSLOduration=1.7344940850000001 podStartE2EDuration="1.734494085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:06.73245738 +0000 UTC m=+707.165221273" watchObservedRunningTime="2026-10-30 07:09:06.734494085 +0000 UTC m=+707.167257977"
Oct 30 07:09:06.820370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:06.820335    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:07.517867 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:09:07.517833    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:09:07.637235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:07.637198    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:07.657934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:07.657902    6738 generic.go:356] "Generic (PLEG): container finished" podID="24b968b3-3fa2-4309-b085-ca6a155b363c" containerID="28bb002af912e63da1d27f28b8c6036947f8f9702e1490087c2c9e3e2b9a33d2" exitCode=0
Oct 30 07:09:07.658351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:07.657941    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v" event={"ID":"24b968b3-3fa2-4309-b085-ca6a155b363c","Type":"ContainerDied","Data":"28bb002af912e63da1d27f28b8c6036947f8f9702e1490087c2c9e3e2b9a33d2"}
Oct 30 07:09:08.798008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.797978    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"]
Oct 30 07:09:08.801417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.801398    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:08.812606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.812586    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:08.821184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.821158    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"]
Oct 30 07:09:08.903607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.903574    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\") pod \"24b968b3-3fa2-4309-b085-ca6a155b363c\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") "
Oct 30 07:09:08.903818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.903735    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\") pod \"24b968b3-3fa2-4309-b085-ca6a155b363c\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") "
Oct 30 07:09:08.903818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.903776    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\" (UniqueName: \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\") pod \"24b968b3-3fa2-4309-b085-ca6a155b363c\" (UID: \"24b968b3-3fa2-4309-b085-ca6a155b363c\") "
Oct 30 07:09:08.903953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.903934    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:08.904147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.904116    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrrj\" (UniqueName: \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:08.904147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.904116    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca" pod "24b968b3-3fa2-4309-b085-ca6a155b363c" (UID: "24b968b3-3fa2-4309-b085-ca6a155b363c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:09:08.904311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.904227    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:08.904468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.904448    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:08.906096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.906073    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm" pod "24b968b3-3fa2-4309-b085-ca6a155b363c" (UID: "24b968b3-3fa2-4309-b085-ca6a155b363c"). InnerVolumeSpecName "kube-api-access-svkjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:09:08.906201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:08.906138    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca" pod "24b968b3-3fa2-4309-b085-ca6a155b363c" (UID: "24b968b3-3fa2-4309-b085-ca6a155b363c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:09:09.005258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005175    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.005427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005265    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.005427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrrj\" (UniqueName: \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.005515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005445    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svkjm\" (UniqueName: \"kubernetes.io/projected/24b968b3-3fa2-4309-b085-ca6a155b363c-kube-api-access-svkjm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:09.005515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005462    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b968b3-3fa2-4309-b085-ca6a155b363c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:09.005930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.005895    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.007624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.007600    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.029381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.029354    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrrj\" (UniqueName: \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\") pod \"maas-api-revoke-sub-c670d859-q685n\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.124777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.124744    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:09.425528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.425500    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"]
Oct 30 07:09:09.426345 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:09:09.426282    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed617b6_6bc6_4ad1_8f01_9d387208397c.slice/crio-d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b WatchSource:0}: Error finding container d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b: Status 404 returned error can't find the container with id d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b
Oct 30 07:09:09.461624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.461584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:09.670334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.670286    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v" event={"ID":"24b968b3-3fa2-4309-b085-ca6a155b363c","Type":"ContainerDied","Data":"c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23"}
Oct 30 07:09:09.670508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.670354    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14b5e64be1ab69e673ece755c13cfe9186e5f867772c6ef21ec37253a9e8f23"
Oct 30 07:09:09.670508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.670284    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"
Oct 30 07:09:09.671734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:09.671708    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n" event={"ID":"3ed617b6-6bc6-4ad1-8f01-9d387208397c","Type":"ContainerStarted","Data":"d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b"}
Oct 30 07:09:10.695313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:10.692944    6738 generic.go:356] "Generic (PLEG): container finished" podID="3ed617b6-6bc6-4ad1-8f01-9d387208397c" containerID="56579fea62f960c19ec50270db245ff2783ccfec8e612de69ffdad7b50cd7c62" exitCode=0
Oct 30 07:09:10.695313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:10.693073    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n" event={"ID":"3ed617b6-6bc6-4ad1-8f01-9d387208397c","Type":"ContainerDied","Data":"56579fea62f960c19ec50270db245ff2783ccfec8e612de69ffdad7b50cd7c62"}
Oct 30 07:09:11.635027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.634986    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:11.901822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.901705    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:11.945636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.945604    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\") pod \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") "
Oct 30 07:09:11.945819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.945673    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\") pod \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") "
Oct 30 07:09:11.945819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.945761    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\" (UniqueName: \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\") pod \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\" (UID: \"3ed617b6-6bc6-4ad1-8f01-9d387208397c\") "
Oct 30 07:09:11.946045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.946015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca" pod "3ed617b6-6bc6-4ad1-8f01-9d387208397c" (UID: "3ed617b6-6bc6-4ad1-8f01-9d387208397c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:09:11.947783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.947761    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca" pod "3ed617b6-6bc6-4ad1-8f01-9d387208397c" (UID: "3ed617b6-6bc6-4ad1-8f01-9d387208397c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:09:11.947894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:11.947878    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj" pod "3ed617b6-6bc6-4ad1-8f01-9d387208397c" (UID: "3ed617b6-6bc6-4ad1-8f01-9d387208397c"). InnerVolumeSpecName "kube-api-access-nfrrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:09:12.047537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.047502    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrrj\" (UniqueName: \"kubernetes.io/projected/3ed617b6-6bc6-4ad1-8f01-9d387208397c-kube-api-access-nfrrj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:12.047537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.047532    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:12.047537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.047541    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed617b6-6bc6-4ad1-8f01-9d387208397c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:12.744834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.744781    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n" event={"ID":"3ed617b6-6bc6-4ad1-8f01-9d387208397c","Type":"ContainerDied","Data":"d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b"}
Oct 30 07:09:12.744834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.744824    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"
Oct 30 07:09:12.745092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:12.744851    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d14c7efaf41d34922081332987317e2c934dd5d18ff3530081c45ef6a395c02b"
Oct 30 07:09:13.536794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:13.536755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:15.641903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:15.641867    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:15.821085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:15.821047    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:15.834186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:15.834159    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:16.781633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:16.781601    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:09:17.461704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:17.461660    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:19.147109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147069    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"]
Oct 30 07:09:19.147803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147782    6738 cpu_manager.go:414] "containerMap: removing container" podUID="24b968b3-3fa2-4309-b085-ca6a155b363c" containerName="revoke-keys"
Oct 30 07:09:19.147895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147804    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b968b3-3fa2-4309-b085-ca6a155b363c" containerName="revoke-keys"
Oct 30 07:09:19.147895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147823    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3ed617b6-6bc6-4ad1-8f01-9d387208397c" containerName="revoke-keys"
Oct 30 07:09:19.147895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147830    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed617b6-6bc6-4ad1-8f01-9d387208397c" containerName="revoke-keys"
Oct 30 07:09:19.148049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147979    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b968b3-3fa2-4309-b085-ca6a155b363c" containerName="revoke-keys"
Oct 30 07:09:19.148049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.147994    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed617b6-6bc6-4ad1-8f01-9d387208397c" containerName="revoke-keys"
Oct 30 07:09:19.151826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.151803    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.169127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.169095    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"]
Oct 30 07:09:19.178889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.178863    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:09:19.179146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.179120    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:09:19.179227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.179148    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:09:19.180012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.179991    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:09:19.241484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.241444    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:19.241755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.241735    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.241800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.241767    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.241921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.241909    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9st7r\" (UniqueName: \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.343333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.343285    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9st7r\" (UniqueName: \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.343524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.343377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.343524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.343400    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.343991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.343973    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.345965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.345936    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.367121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.367089    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9st7r\" (UniqueName: \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\") pod \"maas-api-revoke-sub-e27759a8-rhwk2\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.467391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.467274    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:19.770091 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:09:19.770047    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e52924_91bd_42c3_bbad_1c5a8c631665.slice/crio-9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22 WatchSource:0}: Error finding container 9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22: Status 404 returned error can't find the container with id 9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22
Oct 30 07:09:19.770794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.770762    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"]
Oct 30 07:09:19.791781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:19.791746    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2" event={"ID":"b2e52924-91bd-42c3-bbad-1c5a8c631665","Type":"ContainerStarted","Data":"9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22"}
Oct 30 07:09:20.648657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:20.648598    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:20.809320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:20.809140    6738 generic.go:356] "Generic (PLEG): container finished" podID="b2e52924-91bd-42c3-bbad-1c5a8c631665" containerID="b03ff548d872faf864a0ded649c207e2faaca3ea1cb8edf5889f9e55f52f6422" exitCode=0
Oct 30 07:09:20.810060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:20.809971    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2" event={"ID":"b2e52924-91bd-42c3-bbad-1c5a8c631665","Type":"ContainerDied","Data":"b03ff548d872faf864a0ded649c207e2faaca3ea1cb8edf5889f9e55f52f6422"}
Oct 30 07:09:21.165962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:21.165922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:21.951893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:21.951867    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:22.086016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.085921    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\") pod \"b2e52924-91bd-42c3-bbad-1c5a8c631665\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") "
Oct 30 07:09:22.086016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.085977    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\" (UniqueName: \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\") pod \"b2e52924-91bd-42c3-bbad-1c5a8c631665\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") "
Oct 30 07:09:22.086016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.086005    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\") pod \"b2e52924-91bd-42c3-bbad-1c5a8c631665\" (UID: \"b2e52924-91bd-42c3-bbad-1c5a8c631665\") "
Oct 30 07:09:22.086479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.086454    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca" pod "b2e52924-91bd-42c3-bbad-1c5a8c631665" (UID: "b2e52924-91bd-42c3-bbad-1c5a8c631665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:09:22.088030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.088010    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca" pod "b2e52924-91bd-42c3-bbad-1c5a8c631665" (UID: "b2e52924-91bd-42c3-bbad-1c5a8c631665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:09:22.088122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.088032    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r" pod "b2e52924-91bd-42c3-bbad-1c5a8c631665" (UID: "b2e52924-91bd-42c3-bbad-1c5a8c631665"). InnerVolumeSpecName "kube-api-access-9st7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:09:22.187110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.187076    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:22.187110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.187103    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9st7r\" (UniqueName: \"kubernetes.io/projected/b2e52924-91bd-42c3-bbad-1c5a8c631665-kube-api-access-9st7r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:22.187110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.187114    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e52924-91bd-42c3-bbad-1c5a8c631665-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:09:22.821786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.821756    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"
Oct 30 07:09:22.821984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.821762    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2" event={"ID":"b2e52924-91bd-42c3-bbad-1c5a8c631665","Type":"ContainerDied","Data":"9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22"}
Oct 30 07:09:22.821984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:22.821877    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9412811dd5feac614de09a5184684d623b00c05fb0ba163c92b7bdc04f610b22"
Oct 30 07:09:23.050267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:23.050234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:24.152867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:24.152824    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:25.010688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:25.010657    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:26.854620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:26.854584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:28.357364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:28.357327    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:28.748073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:28.748034    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:30.659810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:30.659776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:30.817640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:30.817602    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:30.943208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:30.943105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:32.536557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:32.536511    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:34.545877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:34.545839    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:36.339077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:36.339032    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:37.518262 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:09:37.518222    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:09:38.214308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:38.214247    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:40.047286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:40.047253    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:41.830820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.830786    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:09:41.831289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.831258    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b2e52924-91bd-42c3-bbad-1c5a8c631665" containerName="revoke-keys"
Oct 30 07:09:41.831289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.831268    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e52924-91bd-42c3-bbad-1c5a8c631665" containerName="revoke-keys"
Oct 30 07:09:41.831454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.831406    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e52924-91bd-42c3-bbad-1c5a8c631665" containerName="revoke-keys"
Oct 30 07:09:41.835332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.835309    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.844209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.844174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:41.854094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.854063    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:09:41.862849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.862819    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"e2e-conflict-b-2e055bbe-kserve-self-signed-certs\""
Oct 30 07:09:41.934345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934285    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.934345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934349    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.934613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934480    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m676q\" (UniqueName: \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.934613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934516    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.934613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934569    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.934747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.934607    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:41.980692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:41.980653    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:42.050822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.050760    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051268    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051423    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m676q\" (UniqueName: \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051542    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051709    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.051870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.051852    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.053468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.053012    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.054353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.054199    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.083383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.083091    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.085022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.084990    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.099114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.099082    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m676q\" (UniqueName: \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\") pod \"e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") " pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.148331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.148288    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:42.449961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.449929    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:09:42.451184 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:09:42.451152    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f1597c3_6f0c_4845_9c3d_b1ebdd5335f9.slice/crio-3b90413b37d1d6b4233b7a4829e5d7c0b008f27cace0f5c67caf4a6488aabf81 WatchSource:0}: Error finding container 3b90413b37d1d6b4233b7a4829e5d7c0b008f27cace0f5c67caf4a6488aabf81: Status 404 returned error can't find the container with id 3b90413b37d1d6b4233b7a4829e5d7c0b008f27cace0f5c67caf4a6488aabf81
Oct 30 07:09:42.940443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.940409    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" event={"ID":"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9","Type":"ContainerStarted","Data":"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"}
Oct 30 07:09:42.940443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.940447    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" event={"ID":"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9","Type":"ContainerStarted","Data":"3b90413b37d1d6b4233b7a4829e5d7c0b008f27cace0f5c67caf4a6488aabf81"}
Oct 30 07:09:42.985537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:42.985476    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" podStartSLOduration=1.985460944 podStartE2EDuration="1.985460944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:42.982913653 +0000 UTC m=+743.415677569" watchObservedRunningTime="2026-10-30 07:09:42.985460944 +0000 UTC m=+743.418224837"
Oct 30 07:09:43.149407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:43.149372    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:43.852328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:43.852267    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:45.744555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:45.744508    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:47.652760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:47.652724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:49.736148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:49.736101    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:50.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:50.131514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:51.662270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:51.662228    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:52.148995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:52.148956    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:52.161720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:52.161690    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:53.009805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:53.009774    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:09:53.648309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:53.648258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:55.556927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:55.556892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:57.443750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:57.443714    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:09:59.440344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:09:59.440275    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:00.247167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.247132    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 07:10:00.251756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.251730    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.269783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.269751    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 07:10:00.279420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.279387    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe2ed\"/\"distinct-2-w1-4fe2ed-kserve-self-signed-certs\""
Oct 30 07:10:00.310132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.310094    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:00.370036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370003    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.370246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370044    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.370246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370061    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.370246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370131    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.370452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370360    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqrl\" (UniqueName: \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.370452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.370386    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.476985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.476937    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqrl\" (UniqueName: \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.477603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.477575    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.477851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.477831    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.477975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.477958    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.478123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.478108    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.478783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.478762    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.479000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.478461    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.479114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.478406    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.479222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.479200    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.490385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.486452    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.496067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.496029    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.509033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.508808    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqrl\" (UniqueName: \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\") pod \"distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") " pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.566595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.566563    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:00.877546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:00.877518    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 07:10:00.879709 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:00.879680    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dcfdfac_7675_4576_835a_373478897a9e.slice/crio-ca2ed8ad972e1eeb8481714980f913f8bd023d5019d7a97c963c24f0e8fa7f32 WatchSource:0}: Error finding container ca2ed8ad972e1eeb8481714980f913f8bd023d5019d7a97c963c24f0e8fa7f32: Status 404 returned error can't find the container with id ca2ed8ad972e1eeb8481714980f913f8bd023d5019d7a97c963c24f0e8fa7f32
Oct 30 07:10:01.050348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:01.050280    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" event={"ID":"5dcfdfac-7675-4576-835a-373478897a9e","Type":"ContainerStarted","Data":"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"}
Oct 30 07:10:01.050348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:01.050348    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" event={"ID":"5dcfdfac-7675-4576-835a-373478897a9e","Type":"ContainerStarted","Data":"ca2ed8ad972e1eeb8481714980f913f8bd023d5019d7a97c963c24f0e8fa7f32"}
Oct 30 07:10:01.109887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:01.109833    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podStartSLOduration=1.109819557 podStartE2EDuration="1.109819557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:01.107460168 +0000 UTC m=+761.540224068" watchObservedRunningTime="2026-10-30 07:10:01.109819557 +0000 UTC m=+761.542583450"
Oct 30 07:10:01.309756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:01.309661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:01.567436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:01.567336    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:03.163151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:03.163109    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:03.380172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:03.380135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:04.947120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:04.947085    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:06.676360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:06.676327    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:06.823105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:06.823068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:07.517685 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:10:07.517639    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:10:08.660703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:08.660664    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:10.567275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:10.567237    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:10.579716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:10.579687    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:10.651686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:10.651648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:10.854356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:10.854260    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:10:10.908596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:10.908564    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:11.122137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:11.122114    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:10:12.564629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:12.564590    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:13.010088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.010054    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 07:10:13.015951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.015924    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.040046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.040008    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 07:10:13.141992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.141952    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142047    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142098    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142136    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77xg\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142173    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142331    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142414    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142445    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.142480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.142474    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77xg\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243565    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243618    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243689    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243728    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243868    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.243997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.243982    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244021    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244161    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244214    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244333    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244444    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.244753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.244730    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.246029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.246003    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.246454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.246404    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.267821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.267753    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.267947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.267911    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77xg\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.331101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.331063    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:13.651417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.651381    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 07:10:13.651851 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:13.651827    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe722c8b_006c_476d_a021_1d1df9faa21b.slice/crio-8311f2124b71141092bbb7d6ee49844380326587ca28fca759a7e37272e5b306 WatchSource:0}: Error finding container 8311f2124b71141092bbb7d6ee49844380326587ca28fca759a7e37272e5b306: Status 404 returned error can't find the container with id 8311f2124b71141092bbb7d6ee49844380326587ca28fca759a7e37272e5b306
Oct 30 07:10:13.654111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.654057    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:13.654199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.654141    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:13.654199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:13.654170    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:14.130844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.130808    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" event={"ID":"fe722c8b-006c-476d-a021-1d1df9faa21b","Type":"ContainerStarted","Data":"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"}
Oct 30 07:10:14.131031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.130862    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" event={"ID":"fe722c8b-006c-476d-a021-1d1df9faa21b","Type":"ContainerStarted","Data":"8311f2124b71141092bbb7d6ee49844380326587ca28fca759a7e37272e5b306"}
Oct 30 07:10:14.178401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.178319    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podStartSLOduration=2.178284862 podStartE2EDuration="2.178284862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:14.177125821 +0000 UTC m=+774.609889714" watchObservedRunningTime="2026-10-30 07:10:14.178284862 +0000 UTC m=+774.611048952"
Oct 30 07:10:14.331200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.331168    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:14.332953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.332923    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:14.333105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.332982    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:14.455535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:14.455457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:15.332508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:15.332471    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:15.333111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:15.332545    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:16.331926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:16.331890    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:16.332123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:16.331957    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:16.538031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:16.537990    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:17.332073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:17.332039    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:17.332258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:17.332100    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:18.333451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:18.332646    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:18.333451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:18.332704    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:18.514199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:18.514153    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:19.331750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:19.331714    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:19.331961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:19.331795    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:20.332221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:20.332181    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:10:20.332770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:20.332246    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 07:10:20.592776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:20.592672    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:22.313714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.313680    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:22.314218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.314023    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:22.314984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.314961    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 07:10:22.435715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.435677    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:22.623453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.623417    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:10:22.674607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:22.674572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:24.339851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:24.339810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:25.875306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:25.875247    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerName="main" containerID="cri-o://b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f" gracePeriod=45
Oct 30 07:10:25.999375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:25.999334    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 07:10:26.142162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.142102    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:10:26.210234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.210198    6738 generic.go:356] "Generic (PLEG): container finished" podID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerID="b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f" exitCode=0
Oct 30 07:10:26.210450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.210256    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" event={"ID":"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9","Type":"ContainerDied","Data":"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"}
Oct 30 07:10:26.210450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.210285    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt" event={"ID":"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9","Type":"ContainerDied","Data":"3b90413b37d1d6b4233b7a4829e5d7c0b008f27cace0f5c67caf4a6488aabf81"}
Oct 30 07:10:26.210450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.210352    6738 scope.go:122] "RemoveContainer" containerID="b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"
Oct 30 07:10:26.210630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.210473    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"
Oct 30 07:10:26.222798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.222770    6738 scope.go:122] "RemoveContainer" containerID="b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"
Oct 30 07:10:26.223163 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:10:26.223140    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f\": container with ID starting with b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f not found: ID does not exist" containerID="b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"
Oct 30 07:10:26.223259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.223174    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f"} err="failed to get container status \"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f\": rpc error: code = NotFound desc = could not find container \"b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f\": container with ID starting with b5718b63b0fcbc0b0b26e5202bc6260915c900a7a88160ea1559b4e85f8eab7f not found: ID does not exist"
Oct 30 07:10:26.235544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235518    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235597    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235695    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235719    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235741    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235762    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\" (UniqueName: \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\") pod \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\" (UID: \"2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9\") "
Oct 30 07:10:26.235820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235777    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:26.235984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235846    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:26.235984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.235936    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:26.236201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.236181    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.236201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.236201    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.236423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.236210    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.237959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.237933    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "kube-api-access-m676q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:26.237959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.237938    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:26.238142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.237983    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs" pod "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" (UID: "2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:26.257341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.257304    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:26.338998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.338955    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.338998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.339000    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.339274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.339020    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m676q\" (UniqueName: \"kubernetes.io/projected/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9-kube-api-access-m676q\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:26.558514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.558474    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:10:26.570814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:26.570780    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-b-2e055bbe-kserve-79ccdb98c7-ll5xt"]
Oct 30 07:10:27.689795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.689760    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" path="/var/lib/kubelet/pods/2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9/volumes"
Oct 30 07:10:27.890766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.890727    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"]
Oct 30 07:10:27.891218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.891206    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerName="main"
Oct 30 07:10:27.891267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.891220    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerName="main"
Oct 30 07:10:27.891337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.891326    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1597c3-6f0c-4845-9c3d-b1ebdd5335f9" containerName="main"
Oct 30 07:10:27.896407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.896383    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:27.915231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.915196    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"]
Oct 30 07:10:27.923172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.923145    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:10:27.923519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.923502    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:10:27.923519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.923511    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:10:27.923681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.923533    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:10:27.959036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.958956    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:27.959036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.959012    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l429h\" (UniqueName: \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:27.959220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:27.959138    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.036093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.036056    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:28.060109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.060072    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.060265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.060145    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l429h\" (UniqueName: \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.060265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.060214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.060965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.060937    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.062637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.062618    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.136899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.136860    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l429h\" (UniqueName: \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\") pod \"maas-api-revoke-sub-508552a7-bhgbt\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.210424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.210334    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:28.491980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.491886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:28.513780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:28.513753    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"]
Oct 30 07:10:28.514774 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:28.514746    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcde54c_2656_4870_abda_67b1d81d15d9.slice/crio-d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93 WatchSource:0}: Error finding container d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93: Status 404 returned error can't find the container with id d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93
Oct 30 07:10:29.240917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:29.240868    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt" event={"ID":"ebcde54c-2656-4870-abda-67b1d81d15d9","Type":"ContainerStarted","Data":"001ddbc14c52fcaf8eedf2b37ebb78fcf072da11198605fa6048265414188099"}
Oct 30 07:10:29.241464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:29.240942    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt" event={"ID":"ebcde54c-2656-4870-abda-67b1d81d15d9","Type":"ContainerStarted","Data":"d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93"}
Oct 30 07:10:30.193696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:30.193654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:30.268109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:30.267058    6738 generic.go:356] "Generic (PLEG): container finished" podID="ebcde54c-2656-4870-abda-67b1d81d15d9" containerID="001ddbc14c52fcaf8eedf2b37ebb78fcf072da11198605fa6048265414188099" exitCode=0
Oct 30 07:10:30.268109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:30.267110    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt" event={"ID":"ebcde54c-2656-4870-abda-67b1d81d15d9","Type":"ContainerDied","Data":"001ddbc14c52fcaf8eedf2b37ebb78fcf072da11198605fa6048265414188099"}
Oct 30 07:10:31.417145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.417113    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:31.518114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.518076    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\") pod \"ebcde54c-2656-4870-abda-67b1d81d15d9\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") "
Oct 30 07:10:31.518318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.518149    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\") pod \"ebcde54c-2656-4870-abda-67b1d81d15d9\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") "
Oct 30 07:10:31.518318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.518199    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\" (UniqueName: \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\") pod \"ebcde54c-2656-4870-abda-67b1d81d15d9\" (UID: \"ebcde54c-2656-4870-abda-67b1d81d15d9\") "
Oct 30 07:10:31.518652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.518623    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca" pod "ebcde54c-2656-4870-abda-67b1d81d15d9" (UID: "ebcde54c-2656-4870-abda-67b1d81d15d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:31.520396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.520360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca" pod "ebcde54c-2656-4870-abda-67b1d81d15d9" (UID: "ebcde54c-2656-4870-abda-67b1d81d15d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:31.520396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.520386    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h" pod "ebcde54c-2656-4870-abda-67b1d81d15d9" (UID: "ebcde54c-2656-4870-abda-67b1d81d15d9"). InnerVolumeSpecName "kube-api-access-l429h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:31.619370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.619321    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:31.619370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.619364    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebcde54c-2656-4870-abda-67b1d81d15d9-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:31.619370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:31.619376    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l429h\" (UniqueName: \"kubernetes.io/projected/ebcde54c-2656-4870-abda-67b1d81d15d9-kube-api-access-l429h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:32.065811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.065769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:32.279940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.279909    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"
Oct 30 07:10:32.280115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.279942    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt" event={"ID":"ebcde54c-2656-4870-abda-67b1d81d15d9","Type":"ContainerDied","Data":"d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93"}
Oct 30 07:10:32.280115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.279992    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27813f4c3d29e58adbb4e2b82fc9fa9b2199aeb3a34f1e574df9e5514cc9e93"
Oct 30 07:10:32.735685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.735654    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"]
Oct 30 07:10:32.736153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.736140    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ebcde54c-2656-4870-abda-67b1d81d15d9" containerName="revoke-keys"
Oct 30 07:10:32.736200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.736154    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcde54c-2656-4870-abda-67b1d81d15d9" containerName="revoke-keys"
Oct 30 07:10:32.736240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.736231    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcde54c-2656-4870-abda-67b1d81d15d9" containerName="revoke-keys"
Oct 30 07:10:32.739638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.739603    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.754987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.754956    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:10:32.767552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.767516    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:10:32.767745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.767638    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:10:32.769872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.769846    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:10:32.771856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.771828    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"]
Oct 30 07:10:32.841714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.841648    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.841888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.841748    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.841953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.841931    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxgt\" (UniqueName: \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.943234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.943184    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.943234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.943240    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.943498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.943282    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxgt\" (UniqueName: \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.943967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.943945    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.945627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.945603    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:32.966837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:32.966808    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxgt\" (UniqueName: \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\") pod \"maas-api-revoke-sub-f2db9512-qnc2k\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:33.054321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.054196    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:33.363655 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:33.363627    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a042b0_89f3_4466_9ca4_428fa12ba061.slice/crio-19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891 WatchSource:0}: Error finding container 19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891: Status 404 returned error can't find the container with id 19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891
Oct 30 07:10:33.363989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.363965    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"]
Oct 30 07:10:33.677890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.677852    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 07:10:33.684585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.681967    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.709289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.709250    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 07:10:33.712308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.712265    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"default-dockercfg-77htj\""
Oct 30 07:10:33.712658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.712628    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"openshift-service-ca.crt\""
Oct 30 07:10:33.712830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.712686    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"kube-root-ca.crt\""
Oct 30 07:10:33.712830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.712749    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"distinct-2-w2-bb6be4-kserve-self-signed-certs\""
Oct 30 07:10:33.743365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.743331    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:33.854752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.854717    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.854752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.854756    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmxp\" (UniqueName: \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.855021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.854890    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.855021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.854926    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.855191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.855170    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.855253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.855238    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956263    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956369    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmxp\" (UniqueName: \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956519    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956552    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.956958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956936    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.957025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956961    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.957025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.956985    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.958890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.958866    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.959034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.959017    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:33.980197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:33.980167    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmxp\" (UniqueName: \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\") pod \"distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") " pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:34.001346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.001307    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:34.061045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.061012    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:34.293111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.293009    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k" event={"ID":"f1a042b0-89f3-4466-9ca4-428fa12ba061","Type":"ContainerStarted","Data":"ff4f301fd9581be5a819012ae4d81745e290872dfbc0fecc12bdee47582d3595"}
Oct 30 07:10:34.293111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.293094    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k" event={"ID":"f1a042b0-89f3-4466-9ca4-428fa12ba061","Type":"ContainerStarted","Data":"19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891"}
Oct 30 07:10:34.305500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.305468    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 07:10:34.306850 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:34.306823    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4d1700b_5742_4b85_b755_44d4b104b5d1.slice/crio-afa1bd1768ee6c16edc4f612fa718aa76c55d91df98c22c7319a1003cf69bf0a WatchSource:0}: Error finding container afa1bd1768ee6c16edc4f612fa718aa76c55d91df98c22c7319a1003cf69bf0a: Status 404 returned error can't find the container with id afa1bd1768ee6c16edc4f612fa718aa76c55d91df98c22c7319a1003cf69bf0a
Oct 30 07:10:34.343605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:34.343549    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k" podStartSLOduration=2.343533056 podStartE2EDuration="2.343533056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:34.340827949 +0000 UTC m=+794.773591840" watchObservedRunningTime="2026-10-30 07:10:34.343533056 +0000 UTC m=+794.776297312"
Oct 30 07:10:35.300585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.300487    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" event={"ID":"d4d1700b-5742-4b85-b755-44d4b104b5d1","Type":"ContainerStarted","Data":"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"}
Oct 30 07:10:35.300585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.300533    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" event={"ID":"d4d1700b-5742-4b85-b755-44d4b104b5d1","Type":"ContainerStarted","Data":"afa1bd1768ee6c16edc4f612fa718aa76c55d91df98c22c7319a1003cf69bf0a"}
Oct 30 07:10:35.302128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.302101    6738 generic.go:356] "Generic (PLEG): container finished" podID="f1a042b0-89f3-4466-9ca4-428fa12ba061" containerID="ff4f301fd9581be5a819012ae4d81745e290872dfbc0fecc12bdee47582d3595" exitCode=0
Oct 30 07:10:35.302256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.302179    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k" event={"ID":"f1a042b0-89f3-4466-9ca4-428fa12ba061","Type":"ContainerDied","Data":"ff4f301fd9581be5a819012ae4d81745e290872dfbc0fecc12bdee47582d3595"}
Oct 30 07:10:35.345003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.344948    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podStartSLOduration=2.344933068 podStartE2EDuration="2.344933068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:35.342647145 +0000 UTC m=+795.775411036" watchObservedRunningTime="2026-10-30 07:10:35.344933068 +0000 UTC m=+795.777696960"
Oct 30 07:10:35.946457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:35.946417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:36.001716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.001661    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:36.456214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.456182    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:36.508857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.508790    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\") pod \"f1a042b0-89f3-4466-9ca4-428fa12ba061\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") "
Oct 30 07:10:36.509051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.508890    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\") pod \"f1a042b0-89f3-4466-9ca4-428fa12ba061\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") "
Oct 30 07:10:36.509051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.508922    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\" (UniqueName: \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\") pod \"f1a042b0-89f3-4466-9ca4-428fa12ba061\" (UID: \"f1a042b0-89f3-4466-9ca4-428fa12ba061\") "
Oct 30 07:10:36.509258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.509228    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca" pod "f1a042b0-89f3-4466-9ca4-428fa12ba061" (UID: "f1a042b0-89f3-4466-9ca4-428fa12ba061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:36.509498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.509471    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:36.510977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.510947    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca" pod "f1a042b0-89f3-4466-9ca4-428fa12ba061" (UID: "f1a042b0-89f3-4466-9ca4-428fa12ba061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:36.511075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.511015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt" pod "f1a042b0-89f3-4466-9ca4-428fa12ba061" (UID: "f1a042b0-89f3-4466-9ca4-428fa12ba061"). InnerVolumeSpecName "kube-api-access-vfxgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:36.611169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.611071    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a042b0-89f3-4466-9ca4-428fa12ba061-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:36.611169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:36.611116    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfxgt\" (UniqueName: \"kubernetes.io/projected/f1a042b0-89f3-4466-9ca4-428fa12ba061-kube-api-access-vfxgt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:37.316005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.315965    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"
Oct 30 07:10:37.316005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.315994    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k" event={"ID":"f1a042b0-89f3-4466-9ca4-428fa12ba061","Type":"ContainerDied","Data":"19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891"}
Oct 30 07:10:37.316252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.316054    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19aa6679291f471898ee226c639d7c3076da5bace7454006df8455bb5a882891"
Oct 30 07:10:37.518693 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:10:37.518658    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:10:37.647135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.647092    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerName="main" containerID="cri-o://9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427" gracePeriod=45
Oct 30 07:10:37.771256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.771215    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Oct 30 07:10:37.834540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.834487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:37.949406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:37.949377    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:10:38.026819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.026773    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.026847    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.026937    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027140    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027199    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027237    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\" (UniqueName: \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\") pod \"54990866-533d-4911-8fb2-cadc0abb2e4f\" (UID: \"54990866-533d-4911-8fb2-cadc0abb2e4f\") "
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027242    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027278    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:38.027658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.027463    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:38.028895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.028811    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.028895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.028851    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.028895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.028872    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.029342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.029244    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:38.029992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.029958    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "kube-api-access-qb5b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:38.032687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.032659    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm" pod "54990866-533d-4911-8fb2-cadc0abb2e4f" (UID: "54990866-533d-4911-8fb2-cadc0abb2e4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:38.130406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.130368    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54990866-533d-4911-8fb2-cadc0abb2e4f-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.130406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.130402    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb5b5\" (UniqueName: \"kubernetes.io/projected/54990866-533d-4911-8fb2-cadc0abb2e4f-kube-api-access-qb5b5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.130406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.130413    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54990866-533d-4911-8fb2-cadc0abb2e4f-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:38.323693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.323593    6738 generic.go:356] "Generic (PLEG): container finished" podID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerID="9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427" exitCode=0
Oct 30 07:10:38.323693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.323658    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"
Oct 30 07:10:38.323693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.323682    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" event={"ID":"54990866-533d-4911-8fb2-cadc0abb2e4f","Type":"ContainerDied","Data":"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"}
Oct 30 07:10:38.323918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.323728    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8" event={"ID":"54990866-533d-4911-8fb2-cadc0abb2e4f","Type":"ContainerDied","Data":"50ff8134a86374bc082f299a85041e58416d30eb9f3f98decf71e7c013e92b92"}
Oct 30 07:10:38.323918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.323745    6738 scope.go:122] "RemoveContainer" containerID="9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"
Oct 30 07:10:38.334630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.334608    6738 scope.go:122] "RemoveContainer" containerID="9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"
Oct 30 07:10:38.334931 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:10:38.334907    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427\": container with ID starting with 9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427 not found: ID does not exist" containerID="9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"
Oct 30 07:10:38.335010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.334942    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427"} err="failed to get container status \"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427\": rpc error: code = NotFound desc = could not find container \"9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427\": container with ID starting with 9901811a3b27bbdca38302062e94cc8e8f7f675389c83b0765787259f334d427 not found: ID does not exist"
Oct 30 07:10:38.377127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.377090    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:10:38.382976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:38.382945    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/e2e-conflict-a-2e055bbe-kserve-6cbc9474c5-pbrb8"]
Oct 30 07:10:39.190059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190024    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:10:39.190572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190555    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f1a042b0-89f3-4466-9ca4-428fa12ba061" containerName="revoke-keys"
Oct 30 07:10:39.190616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190573    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a042b0-89f3-4466-9ca4-428fa12ba061" containerName="revoke-keys"
Oct 30 07:10:39.190651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190623    6738 cpu_manager.go:414] "containerMap: removing container" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerName="main"
Oct 30 07:10:39.190651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190628    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerName="main"
Oct 30 07:10:39.190707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190695    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a042b0-89f3-4466-9ca4-428fa12ba061" containerName="revoke-keys"
Oct 30 07:10:39.190707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.190704    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" containerName="main"
Oct 30 07:10:39.195469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.195447    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.216756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.216721    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:10:39.242039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.241998    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242064    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242105    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242244    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242303    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242467    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2q57\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242562    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.242672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.242645    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.343817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.343785    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.343993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.343844    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.343993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.343876    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.343993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.343905    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2q57\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.343993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.343963    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344044    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344094    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344128    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344230    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344251    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344288    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344455    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344495    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.344912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.344890    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.346264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.346241    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.346610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.346593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.367901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.367869    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.368076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.368007    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2q57\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.509437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.509342    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:39.690215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.690182    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54990866-533d-4911-8fb2-cadc0abb2e4f" path="/var/lib/kubelet/pods/54990866-533d-4911-8fb2-cadc0abb2e4f/volumes"
Oct 30 07:10:39.742544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.742510    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:39.833454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.832518    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:10:39.835553 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:39.835524    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4256c405_e04d_42d3_b223_345ca45f754f.slice/crio-0b4c7c1044965bb22ed04cb95490e38e4820b3e728cf75866565847766cb34b0 WatchSource:0}: Error finding container 0b4c7c1044965bb22ed04cb95490e38e4820b3e728cf75866565847766cb34b0: Status 404 returned error can't find the container with id 0b4c7c1044965bb22ed04cb95490e38e4820b3e728cf75866565847766cb34b0
Oct 30 07:10:39.838056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.838018    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:39.838170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.838106    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:39.838170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:39.838157    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:40.342097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.342056    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" event={"ID":"4256c405-e04d-42d3-b223-345ca45f754f","Type":"ContainerStarted","Data":"983fa0b37fa94b9d33c48e99d698a4ab9a39ab45f0be3d549f5e05cb4d2849a8"}
Oct 30 07:10:40.342564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.342117    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" event={"ID":"4256c405-e04d-42d3-b223-345ca45f754f","Type":"ContainerStarted","Data":"0b4c7c1044965bb22ed04cb95490e38e4820b3e728cf75866565847766cb34b0"}
Oct 30 07:10:40.393745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.393687    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podStartSLOduration=1.393666958 podStartE2EDuration="1.393666958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:40.392897449 +0000 UTC m=+800.825661344" watchObservedRunningTime="2026-10-30 07:10:40.393666958 +0000 UTC m=+800.826430851"
Oct 30 07:10:40.510039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.510005    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:40.511361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.511336    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:40.511471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.511402    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:40.755491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.755459    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"]
Oct 30 07:10:40.759466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.759446    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.790350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.787070    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:10:40.790350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.789757    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:10:40.790350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.789820    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:10:40.790350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.789914    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:10:40.808324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.802109    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"]
Oct 30 07:10:40.860001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.859964    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.860178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.860058    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.860224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.860212    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zf5d\" (UniqueName: \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.961532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.961489    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zf5d\" (UniqueName: \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.961727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.961603    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.961727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.961664    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.962370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.962347    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.964006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.963979    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:40.995167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:40.995135    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zf5d\" (UniqueName: \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\") pod \"maas-api-revoke-sub-01d62e8a-c4pvj\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:41.092374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:41.089487    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:41.400645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:41.400615    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"]
Oct 30 07:10:41.401582 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:41.401556    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af3d83f_f4f2_4511_b22f_33247823e9c4.slice/crio-9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8 WatchSource:0}: Error finding container 9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8: Status 404 returned error can't find the container with id 9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8
Oct 30 07:10:41.510136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:41.510100    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:41.510444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:41.510156    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:41.696886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:41.696782    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:42.355898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.355857    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj" event={"ID":"6af3d83f-f4f2-4511-b22f-33247823e9c4","Type":"ContainerStarted","Data":"9a5dcae311b4e2a9fbea20237bb707ac614c9cff13507f4b7517cff3219afc0d"}
Oct 30 07:10:42.356066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.355951    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj" event={"ID":"6af3d83f-f4f2-4511-b22f-33247823e9c4","Type":"ContainerStarted","Data":"9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8"}
Oct 30 07:10:42.416083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.416028    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj" podStartSLOduration=2.416013346 podStartE2EDuration="2.416013346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:42.413822776 +0000 UTC m=+802.846586683" watchObservedRunningTime="2026-10-30 07:10:42.416013346 +0000 UTC m=+802.848777240"
Oct 30 07:10:42.510546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.510507    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:42.510736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.510575    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:42.642367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.642309    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 07:10:42.646904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.646880    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.662862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.662829    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"distinct-w3-2eda2c-kserve-self-signed-certs\""
Oct 30 07:10:42.665816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.665773    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 07:10:42.701107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.701067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:42.723531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723501    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.723713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723560    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.723713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723650    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gjr\" (UniqueName: \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.723850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723762    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.723850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723829    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.723929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.723852    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.824774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.824738    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.824959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.824782    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.824959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.824878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.824959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.824947    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.825081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.824980    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gjr\" (UniqueName: \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.825081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.825069    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.825722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.825695    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.825813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.825733    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.825813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.825786    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.837611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.836054    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.840217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.840149    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.863166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.863093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gjr\" (UniqueName: \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\") pod \"distinct-w3-2eda2c-kserve-7749b48c75-gdftq\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:42.962385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:42.961931    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:43.111428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.111394    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"]
Oct 30 07:10:43.117757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.117723    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.133966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.133932    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"]
Oct 30 07:10:43.238678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.238579    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.238678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.238643    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.238866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.238823    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7fx\" (UniqueName: \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.302253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.302225    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 07:10:43.303852 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:43.303818    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f41d75b_e6a7_443e_8608_d1fb7e161e20.slice/crio-2915a1db60e466e62861de123e98dc86e6f686281944e933bba7d2cd0d65965f WatchSource:0}: Error finding container 2915a1db60e466e62861de123e98dc86e6f686281944e933bba7d2cd0d65965f: Status 404 returned error can't find the container with id 2915a1db60e466e62861de123e98dc86e6f686281944e933bba7d2cd0d65965f
Oct 30 07:10:43.339588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.339557    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.339740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.339609    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.339740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.339720    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7fx\" (UniqueName: \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.340469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.340439    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.342119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.342094    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.363719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.363690    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7fx\" (UniqueName: \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\") pod \"maas-api-revoke-sub-9332cfbe-vx9j9\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.364009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.363981    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" event={"ID":"9f41d75b-e6a7-443e-8608-d1fb7e161e20","Type":"ContainerStarted","Data":"2915a1db60e466e62861de123e98dc86e6f686281944e933bba7d2cd0d65965f"}
Oct 30 07:10:43.365970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.365940    6738 generic.go:356] "Generic (PLEG): container finished" podID="6af3d83f-f4f2-4511-b22f-33247823e9c4" containerID="9a5dcae311b4e2a9fbea20237bb707ac614c9cff13507f4b7517cff3219afc0d" exitCode=0
Oct 30 07:10:43.366059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.365981    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj" event={"ID":"6af3d83f-f4f2-4511-b22f-33247823e9c4","Type":"ContainerDied","Data":"9a5dcae311b4e2a9fbea20237bb707ac614c9cff13507f4b7517cff3219afc0d"}
Oct 30 07:10:43.437595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.437561    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:43.509773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.509668    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:43.509773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.509731    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:43.752433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.752396    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:43.807195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:43.807138    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"]
Oct 30 07:10:44.002165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.002115    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:44.020937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.020903    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:44.400763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.400432    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9" event={"ID":"9765c6b5-6f11-4203-92e7-e5d9fcd64281","Type":"ContainerStarted","Data":"d057ba9198accdb37ad779c8975f0ec789695ed8fa02b0df8f79f72b2ef2c0a1"}
Oct 30 07:10:44.400763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.400510    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9" event={"ID":"9765c6b5-6f11-4203-92e7-e5d9fcd64281","Type":"ContainerStarted","Data":"928b9f11c9e3fb5e3b8c943668573b77886b4d51bc69ce1f2707557e7e7b8a40"}
Oct 30 07:10:44.404053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.404018    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" event={"ID":"9f41d75b-e6a7-443e-8608-d1fb7e161e20","Type":"ContainerStarted","Data":"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"}
Oct 30 07:10:44.415671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.415644    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:10:44.506116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.506054    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9" podStartSLOduration=1.506038136 podStartE2EDuration="1.506038136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:44.454718035 +0000 UTC m=+804.887481928" watchObservedRunningTime="2026-10-30 07:10:44.506038136 +0000 UTC m=+804.938802026"
Oct 30 07:10:44.510465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.510438    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:44.510629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.510483    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:44.550362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.549591    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podStartSLOduration=2.549571343 podStartE2EDuration="2.549571343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:44.54874141 +0000 UTC m=+804.981505305" watchObservedRunningTime="2026-10-30 07:10:44.549571343 +0000 UTC m=+804.982335237"
Oct 30 07:10:44.598771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.598738    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:44.657840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.656960    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\") pod \"6af3d83f-f4f2-4511-b22f-33247823e9c4\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") "
Oct 30 07:10:44.657840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.657042    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\") pod \"6af3d83f-f4f2-4511-b22f-33247823e9c4\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") "
Oct 30 07:10:44.657840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.657109    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\" (UniqueName: \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\") pod \"6af3d83f-f4f2-4511-b22f-33247823e9c4\" (UID: \"6af3d83f-f4f2-4511-b22f-33247823e9c4\") "
Oct 30 07:10:44.657840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.657393    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca" pod "6af3d83f-f4f2-4511-b22f-33247823e9c4" (UID: "6af3d83f-f4f2-4511-b22f-33247823e9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:44.657840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.657669    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:44.666124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.660490    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d" pod "6af3d83f-f4f2-4511-b22f-33247823e9c4" (UID: "6af3d83f-f4f2-4511-b22f-33247823e9c4"). InnerVolumeSpecName "kube-api-access-4zf5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:44.666912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.666885    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca" pod "6af3d83f-f4f2-4511-b22f-33247823e9c4" (UID: "6af3d83f-f4f2-4511-b22f-33247823e9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:44.762407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.762365    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3d83f-f4f2-4511-b22f-33247823e9c4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:44.762407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.762402    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zf5d\" (UniqueName: \"kubernetes.io/projected/6af3d83f-f4f2-4511-b22f-33247823e9c4-kube-api-access-4zf5d\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:44.963749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:44.963661    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:45.433194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.433069    6738 generic.go:356] "Generic (PLEG): container finished" podID="9765c6b5-6f11-4203-92e7-e5d9fcd64281" containerID="d057ba9198accdb37ad779c8975f0ec789695ed8fa02b0df8f79f72b2ef2c0a1" exitCode=0
Oct 30 07:10:45.433818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.433772    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9" event={"ID":"9765c6b5-6f11-4203-92e7-e5d9fcd64281","Type":"ContainerDied","Data":"d057ba9198accdb37ad779c8975f0ec789695ed8fa02b0df8f79f72b2ef2c0a1"}
Oct 30 07:10:45.444630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.443923    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"
Oct 30 07:10:45.444630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.443988    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj" event={"ID":"6af3d83f-f4f2-4511-b22f-33247823e9c4","Type":"ContainerDied","Data":"9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8"}
Oct 30 07:10:45.444630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.444027    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e1c990e5e23d2ea53167288e892e88bd64541253f71c373a18130d2ce1e88f8"
Oct 30 07:10:45.536347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.522731    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:45.536347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.522795    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:45.939609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:45.939567    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:46.510108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.510066    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 07:10:46.510373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.510141    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 07:10:46.637313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.637277    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:46.728378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.728250    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\") pod \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") "
Oct 30 07:10:46.740420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.734563    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\") pod \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") "
Oct 30 07:10:46.740420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.734766    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\" (UniqueName: \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\") pod \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\" (UID: \"9765c6b5-6f11-4203-92e7-e5d9fcd64281\") "
Oct 30 07:10:46.740420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.738001    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx" pod "9765c6b5-6f11-4203-92e7-e5d9fcd64281" (UID: "9765c6b5-6f11-4203-92e7-e5d9fcd64281"). InnerVolumeSpecName "kube-api-access-kq7fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:46.740420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.738475    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca" pod "9765c6b5-6f11-4203-92e7-e5d9fcd64281" (UID: "9765c6b5-6f11-4203-92e7-e5d9fcd64281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:46.740420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.739050    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca" pod "9765c6b5-6f11-4203-92e7-e5d9fcd64281" (UID: "9765c6b5-6f11-4203-92e7-e5d9fcd64281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:46.836924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.836883    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7fx\" (UniqueName: \"kubernetes.io/projected/9765c6b5-6f11-4203-92e7-e5d9fcd64281-kube-api-access-kq7fx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:46.836924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.836925    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:46.837182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:46.836942    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9765c6b5-6f11-4203-92e7-e5d9fcd64281-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:47.458468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.458433    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"
Oct 30 07:10:47.458468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.458457    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9" event={"ID":"9765c6b5-6f11-4203-92e7-e5d9fcd64281","Type":"ContainerDied","Data":"928b9f11c9e3fb5e3b8c943668573b77886b4d51bc69ce1f2707557e7e7b8a40"}
Oct 30 07:10:47.458693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.458499    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928b9f11c9e3fb5e3b8c943668573b77886b4d51bc69ce1f2707557e7e7b8a40"
Oct 30 07:10:47.514189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.514158    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:47.514611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.514594    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:47.515541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.515520    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:10:47.856592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:47.856509    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:49.640811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:49.640773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:51.660216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:51.660179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:52.963683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:52.963652    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:52.976802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:52.976773    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:53.512378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:53.512331    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:10:53.725229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:53.725192    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:54.092484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.092444    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"]
Oct 30 07:10:54.092948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.092934    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9765c6b5-6f11-4203-92e7-e5d9fcd64281" containerName="revoke-keys"
Oct 30 07:10:54.092993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.092950    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9765c6b5-6f11-4203-92e7-e5d9fcd64281" containerName="revoke-keys"
Oct 30 07:10:54.092993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.092964    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6af3d83f-f4f2-4511-b22f-33247823e9c4" containerName="revoke-keys"
Oct 30 07:10:54.092993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.092969    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af3d83f-f4f2-4511-b22f-33247823e9c4" containerName="revoke-keys"
Oct 30 07:10:54.093154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.093079    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9765c6b5-6f11-4203-92e7-e5d9fcd64281" containerName="revoke-keys"
Oct 30 07:10:54.093154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.093087    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af3d83f-f4f2-4511-b22f-33247823e9c4" containerName="revoke-keys"
Oct 30 07:10:54.096860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.096833    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.115707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.115651    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"]
Oct 30 07:10:54.116791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.116765    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:10:54.118425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.118401    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:10:54.124247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.124192    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:10:54.124450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.124324    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:10:54.227176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.227141    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.227382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.227321    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.227382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.227357    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6vvm\" (UniqueName: \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.252495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.252454    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:10:54.256707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.256680    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.279623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.279582    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:10:54.328803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.328769    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.328997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.328879    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.328997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.328904    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6vvm\" (UniqueName: \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.329483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.329458    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.331218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.331195    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.353027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.352945    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6vvm\" (UniqueName: \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\") pod \"maas-api-revoke-sub-7d24f105-jwhlr\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.429771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.429736    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.429953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.429786    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.429953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.429804    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.429953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.429910    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkjl\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.430060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.429964    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.430060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.430025    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.430060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.430046    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.430152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.430092    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.430210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.430189    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.443120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.443090    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:54.530882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.530845    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.530903    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.530954    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.530998    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531021    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531065    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkjl\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531144    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531212    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.531358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531243    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.532659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531744    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.532659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.531804    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.532659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.532345    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.532659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.532599    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.533176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.533157    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.534945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.534923    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.536732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.536712    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.555615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.555589    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.555763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.555726    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkjl\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.571788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.571755    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:54.746666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.746633    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"]
Oct 30 07:10:54.747661 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:54.747633    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b34d8a6_bbc7_4bab_82ed_8c475ca753fe.slice/crio-a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c WatchSource:0}: Error finding container a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c: Status 404 returned error can't find the container with id a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c
Oct 30 07:10:54.961117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.961077    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:10:54.961227 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:54.961171    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33570e67_8a59_4734_81e4_579732636039.slice/crio-3121842a0d01145881839f2ffbee34e9e4317f0e3383f45b48f3397e0864eb5a WatchSource:0}: Error finding container 3121842a0d01145881839f2ffbee34e9e4317f0e3383f45b48f3397e0864eb5a: Status 404 returned error can't find the container with id 3121842a0d01145881839f2ffbee34e9e4317f0e3383f45b48f3397e0864eb5a
Oct 30 07:10:54.963803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.963769    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:54.963896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.963832    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:54.963896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:54.963861    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:55.135675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.135640    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:55.526506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.526469    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" event={"ID":"33570e67-8a59-4734-81e4-579732636039","Type":"ContainerStarted","Data":"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"}
Oct 30 07:10:55.526715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.526531    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" event={"ID":"33570e67-8a59-4734-81e4-579732636039","Type":"ContainerStarted","Data":"3121842a0d01145881839f2ffbee34e9e4317f0e3383f45b48f3397e0864eb5a"}
Oct 30 07:10:55.528106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.528079    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr" event={"ID":"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe","Type":"ContainerStarted","Data":"7c056e954a7e2fc63a7ede145c60d88367c78765cc1cb57e23a92d3e5fcf304f"}
Oct 30 07:10:55.528216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.528124    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr" event={"ID":"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe","Type":"ContainerStarted","Data":"a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c"}
Oct 30 07:10:55.586326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.576357    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:10:55.624980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.624921    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr" podStartSLOduration=1.624905843 podStartE2EDuration="1.624905843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:55.623074267 +0000 UTC m=+816.055838160" watchObservedRunningTime="2026-10-30 07:10:55.624905843 +0000 UTC m=+816.057669735"
Oct 30 07:10:55.625505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.625481    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podStartSLOduration=1.625474594 podStartE2EDuration="1.625474594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:55.576087242 +0000 UTC m=+816.008851137" watchObservedRunningTime="2026-10-30 07:10:55.625474594 +0000 UTC m=+816.058238486"
Oct 30 07:10:55.752681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:55.752646    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:56.335833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.335799    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"]
Oct 30 07:10:56.345528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.345498    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.359106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.359066    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"]
Oct 30 07:10:56.451015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.450968    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.451175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.451085    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.451241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.451190    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jbz\" (UniqueName: \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.536732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.536693    6738 generic.go:356] "Generic (PLEG): container finished" podID="9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" containerID="7c056e954a7e2fc63a7ede145c60d88367c78765cc1cb57e23a92d3e5fcf304f" exitCode=0
Oct 30 07:10:56.536914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.536775    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr" event={"ID":"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe","Type":"ContainerDied","Data":"7c056e954a7e2fc63a7ede145c60d88367c78765cc1cb57e23a92d3e5fcf304f"}
Oct 30 07:10:56.552641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.552614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jbz\" (UniqueName: \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.552731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.552715    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.552793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.552780    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.553419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.553397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.555217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.555188    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.593197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.593073    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jbz\" (UniqueName: \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\") pod \"maas-api-revoke-sub-69024c76-lxd8g\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:56.595103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.595051    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:10:56.595365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.595111    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:10:56.664177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:56.664142    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:57.051103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.051068    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"]
Oct 30 07:10:57.544482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.544442    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g" event={"ID":"e157f459-7711-4935-9143-0e44b7f386e3","Type":"ContainerStarted","Data":"5ec1db34cb937b03cf8f532916af362e6d40557aca0ea7a7dbc20a6620a33fad"}
Oct 30 07:10:57.545021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.544509    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g" event={"ID":"e157f459-7711-4935-9143-0e44b7f386e3","Type":"ContainerStarted","Data":"83f42f4db08894d909ffffc68d6c7215f3be3389293b14f641753f4f394181c3"}
Oct 30 07:10:57.594529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.591778    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g" podStartSLOduration=1.591760153 podStartE2EDuration="1.591760153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:57.590696056 +0000 UTC m=+818.023459950" watchObservedRunningTime="2026-10-30 07:10:57.591760153 +0000 UTC m=+818.024524046"
Oct 30 07:10:57.595403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.594936    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:10:57.595403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.594990    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:10:57.735219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.735194    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:57.802774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.802681    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\") pod \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") "
Oct 30 07:10:57.802930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.802775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:57.802930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.802795    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\") pod \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") "
Oct 30 07:10:57.802930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.802866    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\" (UniqueName: \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\") pod \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\" (UID: \"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe\") "
Oct 30 07:10:57.803556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.803523    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca" pod "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" (UID: "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:57.805482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.805443    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca" pod "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" (UID: "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:57.807499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.807468    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm" pod "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" (UID: "9b34d8a6-bbc7-4bab-82ed-8c475ca753fe"). InnerVolumeSpecName "kube-api-access-c6vvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:57.904620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.904580    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:57.904620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.904612    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:57.904620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:57.904622    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6vvm\" (UniqueName: \"kubernetes.io/projected/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe-kube-api-access-c6vvm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:10:58.552244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.552154    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"
Oct 30 07:10:58.552244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.552182    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr" event={"ID":"9b34d8a6-bbc7-4bab-82ed-8c475ca753fe","Type":"ContainerDied","Data":"a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c"}
Oct 30 07:10:58.552244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.552228    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b47a447ff6763b811319c73fc0a0c40b96889e34956eaf61858970b9047a8c"
Oct 30 07:10:58.553877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.553854    6738 generic.go:356] "Generic (PLEG): container finished" podID="e157f459-7711-4935-9143-0e44b7f386e3" containerID="5ec1db34cb937b03cf8f532916af362e6d40557aca0ea7a7dbc20a6620a33fad" exitCode=0
Oct 30 07:10:58.553992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.553918    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g" event={"ID":"e157f459-7711-4935-9143-0e44b7f386e3","Type":"ContainerDied","Data":"5ec1db34cb937b03cf8f532916af362e6d40557aca0ea7a7dbc20a6620a33fad"}
Oct 30 07:10:58.587781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.587747    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:10:58.588078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.588052    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:10:58.710051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.710016    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:10:58.710695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.710670    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" containerName="revoke-keys"
Oct 30 07:10:58.710695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.710694    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" containerName="revoke-keys"
Oct 30 07:10:58.710895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.710782    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" containerName="revoke-keys"
Oct 30 07:10:58.714185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.714163    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.747867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.740314    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:10:58.817965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.817878    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.817965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.817916    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818064    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjh7h\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818117    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818136    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818169    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818271    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818365    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.818460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.818389    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.919811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919772    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjh7h\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919829    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919848    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919863    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919894    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919938    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.919956    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920025    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920250    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920254    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920606    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920677    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.920898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.920878    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.922460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.922431    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.922591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.922573    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.944153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.944117    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:58.944366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:58.944341    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjh7h\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:59.031546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.031458    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:10:59.393914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.393869    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:10:59.399915 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:10:59.399882    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0608df_6277_4846_bdcb_40ce59f7d918.slice/crio-8f8e6deb4af02b67b16db19eaef492fac34c4a4e2a1f911c9b0b3e2d7e2b7f40 WatchSource:0}: Error finding container 8f8e6deb4af02b67b16db19eaef492fac34c4a4e2a1f911c9b0b3e2d7e2b7f40: Status 404 returned error can't find the container with id 8f8e6deb4af02b67b16db19eaef492fac34c4a4e2a1f911c9b0b3e2d7e2b7f40
Oct 30 07:10:59.402708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.402663    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:59.402788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.402770    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:59.402844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.402819    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:10:59.591328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.587394    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" event={"ID":"7e0608df-6277-4846-bdcb-40ce59f7d918","Type":"ContainerStarted","Data":"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"}
Oct 30 07:10:59.591328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.587513    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" event={"ID":"7e0608df-6277-4846-bdcb-40ce59f7d918","Type":"ContainerStarted","Data":"8f8e6deb4af02b67b16db19eaef492fac34c4a4e2a1f911c9b0b3e2d7e2b7f40"}
Oct 30 07:10:59.591328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.587950    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:10:59.591328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.588008    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:10:59.657661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.657534    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podStartSLOduration=1.6575137930000001 podStartE2EDuration="1.657513793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:59.650841453 +0000 UTC m=+820.083605352" watchObservedRunningTime="2026-10-30 07:10:59.657513793 +0000 UTC m=+820.090277686"
Oct 30 07:10:59.819521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.819486    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:10:59.832586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.832555    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:10:59.934372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.933728    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\") pod \"e157f459-7711-4935-9143-0e44b7f386e3\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") "
Oct 30 07:10:59.934372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.933812    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\") pod \"e157f459-7711-4935-9143-0e44b7f386e3\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") "
Oct 30 07:10:59.934372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.933958    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\" (UniqueName: \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\") pod \"e157f459-7711-4935-9143-0e44b7f386e3\" (UID: \"e157f459-7711-4935-9143-0e44b7f386e3\") "
Oct 30 07:10:59.950425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.948062    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca" pod "e157f459-7711-4935-9143-0e44b7f386e3" (UID: "e157f459-7711-4935-9143-0e44b7f386e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:59.952273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.952242    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca" pod "e157f459-7711-4935-9143-0e44b7f386e3" (UID: "e157f459-7711-4935-9143-0e44b7f386e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:10:59.954629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:10:59.954587    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz" pod "e157f459-7711-4935-9143-0e44b7f386e3" (UID: "e157f459-7711-4935-9143-0e44b7f386e3"). InnerVolumeSpecName "kube-api-access-66jbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:00.031628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.031594    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:11:00.033705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.033673    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:00.033883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.033726    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:00.035218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.035180    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:00.035218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.035210    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e157f459-7711-4935-9143-0e44b7f386e3-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:00.035218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.035221    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jbz\" (UniqueName: \"kubernetes.io/projected/e157f459-7711-4935-9143-0e44b7f386e3-kube-api-access-66jbz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:00.576317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.575028    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:11:00.576317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.575090    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:11:00.624217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.624189    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"
Oct 30 07:11:00.624617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.624229    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g" event={"ID":"e157f459-7711-4935-9143-0e44b7f386e3","Type":"ContainerDied","Data":"83f42f4db08894d909ffffc68d6c7215f3be3389293b14f641753f4f394181c3"}
Oct 30 07:11:00.624617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:00.624285    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f42f4db08894d909ffffc68d6c7215f3be3389293b14f641753f4f394181c3"
Oct 30 07:11:01.031776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:01.031738    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:01.031942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:01.031800    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:01.573781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:01.573743    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 07:11:01.574090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:01.574053    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 07:11:01.880316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:01.876765    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:02.032233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:02.032196    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:02.032456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:02.032249    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:02.622343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:02.622278    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:11:02.622857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:02.622834    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:11:02.624098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:02.624072    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:11:03.032541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:03.032453    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:03.032541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:03.032508    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:04.037858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:04.037812    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:04.038741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:04.038695    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:04.502485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:04.502411    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:04.755536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:04.755424    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:04.770506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:04.770472    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:05.006217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.006114    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:11:05.007463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.006069    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.028320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.019265    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.028577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.028543    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"]
Oct 30 07:11:05.029771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.029745    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e157f459-7711-4935-9143-0e44b7f386e3" containerName="revoke-keys"
Oct 30 07:11:05.029906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.029891    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e157f459-7711-4935-9143-0e44b7f386e3" containerName="revoke-keys"
Oct 30 07:11:05.030160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.030142    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e157f459-7711-4935-9143-0e44b7f386e3" containerName="revoke-keys"
Oct 30 07:11:05.042320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.037515    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:05.042922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.042882    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:05.097339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.053614    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"]
Oct 30 07:11:05.097339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.053751    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.161334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.147562    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.161334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.148374    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:05.161334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.148444    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:05.161334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.148552    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:05.161334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.148660    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:05.248346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.247375    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:11:05.356501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.337286    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.356501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.337473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz8g\" (UniqueName: \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.356501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.337599    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.417386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.417336    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.453055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.453008    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.454277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.454250    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz8g\" (UniqueName: \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.454898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.454873    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.455871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.455775    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.474321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.473818    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.486821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.486742    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz8g\" (UniqueName: \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\") pod \"maas-api-revoke-sub-a9e6b031-sb4v4\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.507854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.507805    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:11:05.509701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.509669    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:11:05.530318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.528722    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:11:05.579661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.579627    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.613053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.612973    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:11:05.633787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.633475    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.786425    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.789994    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791634    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791723    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvgd\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791804    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791830    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791897    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791951    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.791989    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.792113    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.848472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.792143    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916619    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916718    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvgd\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916794    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916830    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916889    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916930    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.916965    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.917070    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.917098    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.917602    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.918263    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.918846    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.926265    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:05.953321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:05.929650    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.036367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.013489    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.036367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.021639    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.036367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.022932    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.077321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.052330    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:11:06.088354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.077971    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:11:06.094332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.088685    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:11:06.094920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.063857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvgd\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.104321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.064799    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:06.109434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.109379    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:06.177321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.141968    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.177321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.150369    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.177321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.165494    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:11:06.330512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.274750    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.330512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.276066    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.330512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.276504    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:11:06.452282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.452247    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:06.492307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.470168    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:11:06.523555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.511402    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.665356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.664472    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:11:06.665356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.664536    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:06.812319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:06.810728    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:11:07.049347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.040005    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:07.122361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.103200    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:07.122361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.103259    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:07.200558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.200511    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:07.289691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.276394    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:07.439468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.439418    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:07.554829 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:07.444476    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d524e8_32b2_4ec4_a763_e11263546f34.slice/crio-0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d WatchSource:0}: Error finding container 0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d: Status 404 returned error can't find the container with id 0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d
Oct 30 07:11:07.554829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.464943    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:11:07.659235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.659203    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:11:07.822447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.822376    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:11:07.859065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.858356    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"]
Oct 30 07:11:07.859065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.858459    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:11:07.859065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.858494    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:11:07.859065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:07.858529    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:11:07.860458 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:11:07.860320    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:11:08.030419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.023556    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:11:08.030419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.026977    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:08.030419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.027072    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:08.030419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.027119    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:08.032749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.032720    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 07:11:08.032873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.032778    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 07:11:08.080364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.080287    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:08.115954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.115918    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4" event={"ID":"55d524e8-32b2-4ec4-a763-e11263546f34","Type":"ContainerStarted","Data":"0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d"}
Oct 30 07:11:08.118232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:08.118191    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" event={"ID":"5ec57481-f6e1-4866-9503-603d9c9bba1b","Type":"ContainerStarted","Data":"d7170e1ca38a2c302fdd43e8d8bf4340f94473c631304b420e3448002823613a"}
Oct 30 07:11:09.128233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.128190    6738 generic.go:356] "Generic (PLEG): container finished" podID="55d524e8-32b2-4ec4-a763-e11263546f34" containerID="e223e85762137cbdc21650cd75906095ac86c3f84c1deccefa21010bdf04d5e1" exitCode=0
Oct 30 07:11:09.128703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.128270    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4" event={"ID":"55d524e8-32b2-4ec4-a763-e11263546f34","Type":"ContainerDied","Data":"e223e85762137cbdc21650cd75906095ac86c3f84c1deccefa21010bdf04d5e1"}
Oct 30 07:11:09.130629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.130597    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" event={"ID":"5ec57481-f6e1-4866-9503-603d9c9bba1b","Type":"ContainerStarted","Data":"d37581fad5a5002c481fe0f4f74118a8fdc9609166f7fcb1fd2df25cdcb59214"}
Oct 30 07:11:09.234867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.234726    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podStartSLOduration=4.234705997 podStartE2EDuration="4.234705997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:09.233659311 +0000 UTC m=+829.666423205" watchObservedRunningTime="2026-10-30 07:11:09.234705997 +0000 UTC m=+829.667469890"
Oct 30 07:11:09.488377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.488231    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:09.494319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.493044    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:09.494319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:09.493107    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:10.088327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.077411    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:11:10.088327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.077489    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:10.088327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.079780    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:10.434718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.434685    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:10.506320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.497720    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:10.506320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.497785    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:10.527073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.527024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\") pod \"55d524e8-32b2-4ec4-a763-e11263546f34\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") "
Oct 30 07:11:10.527288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.527146    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\") pod \"55d524e8-32b2-4ec4-a763-e11263546f34\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") "
Oct 30 07:11:10.527288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.527276    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\" (UniqueName: \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\") pod \"55d524e8-32b2-4ec4-a763-e11263546f34\" (UID: \"55d524e8-32b2-4ec4-a763-e11263546f34\") "
Oct 30 07:11:10.527946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.527918    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca" pod "55d524e8-32b2-4ec4-a763-e11263546f34" (UID: "55d524e8-32b2-4ec4-a763-e11263546f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:10.534821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.534770    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca" pod "55d524e8-32b2-4ec4-a763-e11263546f34" (UID: "55d524e8-32b2-4ec4-a763-e11263546f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:10.534821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.534807    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g" pod "55d524e8-32b2-4ec4-a763-e11263546f34" (UID: "55d524e8-32b2-4ec4-a763-e11263546f34"). InnerVolumeSpecName "kube-api-access-gnz8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:10.629454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.629418    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:10.629454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.629451    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d524e8-32b2-4ec4-a763-e11263546f34-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:10.629454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.629463    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnz8g\" (UniqueName: \"kubernetes.io/projected/55d524e8-32b2-4ec4-a763-e11263546f34-kube-api-access-gnz8g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:10.721769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.721676    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:11:10.722083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.722061    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:11:10.722711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:10.722690    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:11:11.167066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:11.167028    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"
Oct 30 07:11:11.167066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:11.167056    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4" event={"ID":"55d524e8-32b2-4ec4-a763-e11263546f34","Type":"ContainerDied","Data":"0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d"}
Oct 30 07:11:11.167328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:11.167111    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3d6117eebad41be4b7b4e72233fe87e1e9b6aded5004a6d102f599d55df44d"
Oct 30 07:11:11.488762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:11.488665    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:11.489423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:11.489380    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:12.047580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:12.047541    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:12.493166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:12.492662    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:12.493166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:12.492735    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:13.492320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:13.491415    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:13.492320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:13.491485    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:13.979320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:13.967615    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:14.484495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.484452    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 07:11:14.484661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.484527    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 07:11:14.648858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.648814    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"]
Oct 30 07:11:14.649368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.649354    6738 cpu_manager.go:414] "containerMap: removing container" podUID="55d524e8-32b2-4ec4-a763-e11263546f34" containerName="revoke-keys"
Oct 30 07:11:14.649419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.649369    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d524e8-32b2-4ec4-a763-e11263546f34" containerName="revoke-keys"
Oct 30 07:11:14.649482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.649473    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d524e8-32b2-4ec4-a763-e11263546f34" containerName="revoke-keys"
Oct 30 07:11:14.654084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.654061    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.669667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.669632    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:14.674519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.674488    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"]
Oct 30 07:11:14.681961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.681927    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:14.681961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.681935    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:14.682882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.682856    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:14.712428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.712387    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7qp\" (UniqueName: \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.712652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.712572    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.712652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.712646    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.814583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.814482    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7qp\" (UniqueName: \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.814770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.814632    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.814770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.814693    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.815474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.815453    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.817151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.817128    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.838882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.838851    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7qp\" (UniqueName: \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\") pod \"maas-api-revoke-sub-9152df1e-8gmwq\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:14.971568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:14.971527    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:15.280557 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:15.280527    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8287013d_a683_4e0f_b786_8e90b46053de.slice/crio-3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5 WatchSource:0}: Error finding container 3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5: Status 404 returned error can't find the container with id 3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5
Oct 30 07:11:15.282512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:15.282479    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"]
Oct 30 07:11:15.823576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:15.823541    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:15.823919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:15.823890    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:15.834149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:15.834064    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:11:15.943644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:15.943610    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:16.222797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:16.222767    6738 generic.go:356] "Generic (PLEG): container finished" podID="8287013d-a683-4e0f-b786-8e90b46053de" containerID="3998a28c7696f1a0c9b4ae5d5e0773c2bea7158cd351d838cf6372313c9124a1" exitCode=0
Oct 30 07:11:16.222949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:16.222850    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq" event={"ID":"8287013d-a683-4e0f-b786-8e90b46053de","Type":"ContainerDied","Data":"3998a28c7696f1a0c9b4ae5d5e0773c2bea7158cd351d838cf6372313c9124a1"}
Oct 30 07:11:16.222949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:16.222910    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq" event={"ID":"8287013d-a683-4e0f-b786-8e90b46053de","Type":"ContainerStarted","Data":"3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5"}
Oct 30 07:11:17.050479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.050434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:17.456994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.456968    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:17.567480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.567442    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\") pod \"8287013d-a683-4e0f-b786-8e90b46053de\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") "
Oct 30 07:11:17.567691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.567557    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\" (UniqueName: \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\") pod \"8287013d-a683-4e0f-b786-8e90b46053de\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") "
Oct 30 07:11:17.567691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.567638    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\") pod \"8287013d-a683-4e0f-b786-8e90b46053de\" (UID: \"8287013d-a683-4e0f-b786-8e90b46053de\") "
Oct 30 07:11:17.568031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.568000    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca" pod "8287013d-a683-4e0f-b786-8e90b46053de" (UID: "8287013d-a683-4e0f-b786-8e90b46053de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:17.568160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.568132    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:17.569683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.569652    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca" pod "8287013d-a683-4e0f-b786-8e90b46053de" (UID: "8287013d-a683-4e0f-b786-8e90b46053de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:17.570098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.570079    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp" pod "8287013d-a683-4e0f-b786-8e90b46053de" (UID: "8287013d-a683-4e0f-b786-8e90b46053de"). InnerVolumeSpecName "kube-api-access-dc7qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:17.669144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.669103    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287013d-a683-4e0f-b786-8e90b46053de-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:17.669144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:17.669133    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7qp\" (UniqueName: \"kubernetes.io/projected/8287013d-a683-4e0f-b786-8e90b46053de-kube-api-access-dc7qp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:18.044125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.044027    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:18.293278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.293251    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"
Oct 30 07:11:18.293722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.293304    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq" event={"ID":"8287013d-a683-4e0f-b786-8e90b46053de","Type":"ContainerDied","Data":"3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5"}
Oct 30 07:11:18.293722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.293354    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3384a4a9bd1ccd2fc69e0c37aee8e1af7750949915397e4676a07c9b35d47eb5"
Oct 30 07:11:18.921582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.921545    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"]
Oct 30 07:11:18.922241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.922218    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8287013d-a683-4e0f-b786-8e90b46053de" containerName="revoke-keys"
Oct 30 07:11:18.922241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.922240    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8287013d-a683-4e0f-b786-8e90b46053de" containerName="revoke-keys"
Oct 30 07:11:18.922447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.922355    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8287013d-a683-4e0f-b786-8e90b46053de" containerName="revoke-keys"
Oct 30 07:11:18.925966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.925949    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:18.945694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.945663    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"]
Oct 30 07:11:18.953264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.953234    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:18.953264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.953234    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:18.953473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.953246    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:18.953473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:18.953304    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:19.088243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.088207    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.088460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.088386    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.088460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.088420    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xndg8\" (UniqueName: \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.189261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.189166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.189453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.189281    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.189453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.189317    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xndg8\" (UniqueName: \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.189949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.189928    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.191501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.191481    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.213481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.213450    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xndg8\" (UniqueName: \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\") pod \"maas-api-revoke-sub-8d31722c-9dsgx\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.240191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.240151    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:19.549826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.549796    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"]
Oct 30 07:11:19.550170 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:19.550145    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20807347_b99f_4f72_8899_ecdf8002630a.slice/crio-7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5 WatchSource:0}: Error finding container 7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5: Status 404 returned error can't find the container with id 7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5
Oct 30 07:11:19.864338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:19.864196    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:20.311129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:20.311094    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx" event={"ID":"20807347-b99f-4f72-8899-ecdf8002630a","Type":"ContainerStarted","Data":"6aa2895b02a1e73ee3fb631e3a43b2f2be5e98470fb6ecb0992a1ec9be413167"}
Oct 30 07:11:20.311371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:20.311148    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx" event={"ID":"20807347-b99f-4f72-8899-ecdf8002630a","Type":"ContainerStarted","Data":"7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5"}
Oct 30 07:11:20.373467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:20.373411    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx" podStartSLOduration=2.373395456 podStartE2EDuration="2.373395456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:20.37136026 +0000 UTC m=+840.804124152" watchObservedRunningTime="2026-10-30 07:11:20.373395456 +0000 UTC m=+840.806159346"
Oct 30 07:11:21.319804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:21.319764    6738 generic.go:356] "Generic (PLEG): container finished" podID="20807347-b99f-4f72-8899-ecdf8002630a" containerID="6aa2895b02a1e73ee3fb631e3a43b2f2be5e98470fb6ecb0992a1ec9be413167" exitCode=0
Oct 30 07:11:21.320248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:21.319849    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx" event={"ID":"20807347-b99f-4f72-8899-ecdf8002630a","Type":"ContainerDied","Data":"6aa2895b02a1e73ee3fb631e3a43b2f2be5e98470fb6ecb0992a1ec9be413167"}
Oct 30 07:11:21.753779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:21.753743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:22.480418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.480389    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:22.541448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.541412    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\") pod \"20807347-b99f-4f72-8899-ecdf8002630a\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") "
Oct 30 07:11:22.541645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.541624    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\" (UniqueName: \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\") pod \"20807347-b99f-4f72-8899-ecdf8002630a\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") "
Oct 30 07:11:22.541719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.541703    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\") pod \"20807347-b99f-4f72-8899-ecdf8002630a\" (UID: \"20807347-b99f-4f72-8899-ecdf8002630a\") "
Oct 30 07:11:22.541873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.541843    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca" pod "20807347-b99f-4f72-8899-ecdf8002630a" (UID: "20807347-b99f-4f72-8899-ecdf8002630a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:22.542236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.542206    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:22.543877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.543847    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8" pod "20807347-b99f-4f72-8899-ecdf8002630a" (UID: "20807347-b99f-4f72-8899-ecdf8002630a"). InnerVolumeSpecName "kube-api-access-xndg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:22.543973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.543888    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca" pod "20807347-b99f-4f72-8899-ecdf8002630a" (UID: "20807347-b99f-4f72-8899-ecdf8002630a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:22.643960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.643920    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xndg8\" (UniqueName: \"kubernetes.io/projected/20807347-b99f-4f72-8899-ecdf8002630a-kube-api-access-xndg8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:22.643960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:22.643954    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20807347-b99f-4f72-8899-ecdf8002630a-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:23.373391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:23.370632    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx" event={"ID":"20807347-b99f-4f72-8899-ecdf8002630a","Type":"ContainerDied","Data":"7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5"}
Oct 30 07:11:23.373391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:23.370702    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba5e892c82c140229bd6d61628dcad177b816974ffc75c8d474c2f9cd24b8f5"
Oct 30 07:11:23.373391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:23.370781    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"
Oct 30 07:11:24.419319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.418344    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:24.517319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.516814    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:24.725675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.711924    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:24.801318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.800404    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:11:24.801318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.800472    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:24.914316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.913406    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:24.996047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:24.995967    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:26.562121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.562078    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"]
Oct 30 07:11:26.562865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.562842    6738 cpu_manager.go:414] "containerMap: removing container" podUID="20807347-b99f-4f72-8899-ecdf8002630a" containerName="revoke-keys"
Oct 30 07:11:26.562963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.562867    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20807347-b99f-4f72-8899-ecdf8002630a" containerName="revoke-keys"
Oct 30 07:11:26.563020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.563001    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="20807347-b99f-4f72-8899-ecdf8002630a" containerName="revoke-keys"
Oct 30 07:11:26.566777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.566756    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.583343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.583316    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:26.583343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.583285    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:26.583565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.583289    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:26.592147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.592108    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptg7v\" (UniqueName: \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.592376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.592351    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.592522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.592500    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.595978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.595940    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:26.599742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.599714    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"]
Oct 30 07:11:26.694114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.694076    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptg7v\" (UniqueName: \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.694357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.694162    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.694357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.694219    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.694949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.694929    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.696499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.696479    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.718342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.718309    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptg7v\" (UniqueName: \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\") pod \"maas-api-revoke-sub-265191bd-gmfv6\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:26.880865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:26.880833    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:27.231373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:27.229124    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"]
Oct 30 07:11:27.241939 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:27.241618    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db96ea6_8f2e_4f8b_b19e_8a1e86155a93.slice/crio-d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322 WatchSource:0}: Error finding container d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322: Status 404 returned error can't find the container with id d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322
Oct 30 07:11:27.249746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:27.249716    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:27.414567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:27.414528    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6" event={"ID":"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93","Type":"ContainerStarted","Data":"d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322"}
Oct 30 07:11:28.080495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.080456    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"]
Oct 30 07:11:28.084900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.084868    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.105010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.104978    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"]
Oct 30 07:11:28.113765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.113736    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.114042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.114016    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfshm\" (UniqueName: \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.114138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.114118    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.215621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.215588    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfshm\" (UniqueName: \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.215810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.215646    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.215810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.215684    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.216487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.216461    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.218079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.218058    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.239914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.239879    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfshm\" (UniqueName: \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\") pod \"maas-api-revoke-sub-92defb9e-gcl6b\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.408275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.408238    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:28.428617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.428586    6738 generic.go:356] "Generic (PLEG): container finished" podID="0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" containerID="7163acfe7f72d0e59ee5265369b47411a7b1c7fc12098d48308d2dde79be2734" exitCode=0
Oct 30 07:11:28.428737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.428671    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6" event={"ID":"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93","Type":"ContainerDied","Data":"7163acfe7f72d0e59ee5265369b47411a7b1c7fc12098d48308d2dde79be2734"}
Oct 30 07:11:28.631992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.631956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:28.716385 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:28.716346    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f86e2d_7f2c_477d_af3b_c60781b10626.slice/crio-9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a WatchSource:0}: Error finding container 9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a: Status 404 returned error can't find the container with id 9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a
Oct 30 07:11:28.717853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.717824    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"]
Oct 30 07:11:28.806121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.806087    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:11:28.811451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.811426    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.826955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.826923    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.826963    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827007    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjrd\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827117    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827169    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827253    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827274    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827327    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.827455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.827348    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.849244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.849200    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:11:28.929031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.928989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929072    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929092    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929123    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929141    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929225    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929254    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929337    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjrd\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929383    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.929697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.929656    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.930074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.930044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.930194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.930044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.930194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.930049    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.930194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.930124    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.931596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.931569    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.931702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.931646    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.952909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.952874    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:28.953035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:28.953011    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjrd\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:29.051824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.051781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:29.136502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.133644    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:29.372237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.372210    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"]
Oct 30 07:11:29.378470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.378447    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.395453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.395419    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"]
Oct 30 07:11:29.438951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.438917    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b" event={"ID":"e1f86e2d-7f2c-477d-af3b-c60781b10626","Type":"ContainerStarted","Data":"a26fdecf902cd94521ccce9ef8ff5a80c94d771c2637933455134efa7e56c75d"}
Oct 30 07:11:29.439148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.438979    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b" event={"ID":"e1f86e2d-7f2c-477d-af3b-c60781b10626","Type":"ContainerStarted","Data":"9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a"}
Oct 30 07:11:29.445487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.445453    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.445657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.445637    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsclm\" (UniqueName: \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.445796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.445775    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.484589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.480810    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:11:29.484589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.484039    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b" podStartSLOduration=1.484024275 podStartE2EDuration="1.484024275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:29.479213196 +0000 UTC m=+849.911977091" watchObservedRunningTime="2026-10-30 07:11:29.484024275 +0000 UTC m=+849.916788167"
Oct 30 07:11:29.484589 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:29.484378    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0085f43_d1ed_4f05_b6ed_aeffdf20cf44.slice/crio-f538b4632a2fb1de94e57402566869894f84fab39faac0a944fefed1529813c6 WatchSource:0}: Error finding container f538b4632a2fb1de94e57402566869894f84fab39faac0a944fefed1529813c6: Status 404 returned error can't find the container with id f538b4632a2fb1de94e57402566869894f84fab39faac0a944fefed1529813c6
Oct 30 07:11:29.488892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.488853    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:29.489046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.488930    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:29.489046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.488963    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:29.547218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.547181    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsclm\" (UniqueName: \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.547439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.547242    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.547439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.547406    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.548285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.548255    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.551918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.551892    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.575477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.575403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsclm\" (UniqueName: \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\") pod \"maas-api-revoke-sub-06163e03-jqp2c\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.598480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.598446    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:29.648964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.648883    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\") pod \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") "
Oct 30 07:11:29.649129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.649058    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\" (UniqueName: \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\") pod \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") "
Oct 30 07:11:29.649129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.649118    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\") pod \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\" (UID: \"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93\") "
Oct 30 07:11:29.649378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.649306    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca" pod "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" (UID: "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:29.649677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.649647    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:29.656443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.656408    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v" pod "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" (UID: "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93"). InnerVolumeSpecName "kube-api-access-ptg7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:29.657968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.657936    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca" pod "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" (UID: "0db96ea6-8f2e-4f8b-b19e-8a1e86155a93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:29.695780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.695747    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:29.751724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.751663    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptg7v\" (UniqueName: \"kubernetes.io/projected/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-kube-api-access-ptg7v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:29.751724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:29.751698    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:30.018521 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:30.018488    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1d9080_84a8_4f99_a935_17a1fefae0bb.slice/crio-aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3 WatchSource:0}: Error finding container aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3: Status 404 returned error can't find the container with id aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3
Oct 30 07:11:30.018712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.018688    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"]
Oct 30 07:11:30.061278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.061251    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 07:11:30.062203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.062179    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" containerName="revoke-keys"
Oct 30 07:11:30.062336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.062205    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" containerName="revoke-keys"
Oct 30 07:11:30.062418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.062383    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" containerName="revoke-keys"
Oct 30 07:11:30.067318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.067285    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.094132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.094100    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"distinct-2-w0-06b26e-kserve-self-signed-certs\""
Oct 30 07:11:30.098592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.095521    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"kube-root-ca.crt\""
Oct 30 07:11:30.100529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.100501    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"openshift-service-ca.crt\""
Oct 30 07:11:30.100787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.100773    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"default-dockercfg-spf2z\""
Oct 30 07:11:30.114677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.114650    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 07:11:30.125344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.125200    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:30.155945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.155915    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.156399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.156089    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5k5\" (UniqueName: \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.156399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.156129    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.156399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.156259    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.156399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.156355    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.156399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.156398    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.262724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.262683    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.277496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.277397    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.277767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.277749    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.277906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.277892    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.278172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.278156    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5k5\" (UniqueName: \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.278319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.278304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.278870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.278813    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.282381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.276177    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.291319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.282841    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.291559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.288695    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.305967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.305933    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.326383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.326347    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5k5\" (UniqueName: \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\") pod \"distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.432582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.432538    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:30.462705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.462246    6738 generic.go:356] "Generic (PLEG): container finished" podID="e1f86e2d-7f2c-477d-af3b-c60781b10626" containerID="a26fdecf902cd94521ccce9ef8ff5a80c94d771c2637933455134efa7e56c75d" exitCode=0
Oct 30 07:11:30.462705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.462397    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b" event={"ID":"e1f86e2d-7f2c-477d-af3b-c60781b10626","Type":"ContainerDied","Data":"a26fdecf902cd94521ccce9ef8ff5a80c94d771c2637933455134efa7e56c75d"}
Oct 30 07:11:30.465611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.464743    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" event={"ID":"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44","Type":"ContainerStarted","Data":"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"}
Oct 30 07:11:30.465611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.464802    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" event={"ID":"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44","Type":"ContainerStarted","Data":"f538b4632a2fb1de94e57402566869894f84fab39faac0a944fefed1529813c6"}
Oct 30 07:11:30.467488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.467416    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c" event={"ID":"1c1d9080-84a8-4f99-a935-17a1fefae0bb","Type":"ContainerStarted","Data":"aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3"}
Oct 30 07:11:30.492180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.492141    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6" event={"ID":"0db96ea6-8f2e-4f8b-b19e-8a1e86155a93","Type":"ContainerDied","Data":"d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322"}
Oct 30 07:11:30.492440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.492423    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87b0aa62dea51c4c1dc0a29111a34522cf8ef5cacb78300c1b657e458d62322"
Oct 30 07:11:30.492636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.492621    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"
Oct 30 07:11:30.611514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.611453    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podStartSLOduration=2.611433611 podStartE2EDuration="2.611433611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:30.592467628 +0000 UTC m=+851.025231521" watchObservedRunningTime="2026-10-30 07:11:30.611433611 +0000 UTC m=+851.044197505"
Oct 30 07:11:30.802568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:30.802536    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 07:11:30.804071 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:30.804037    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb64ba02_617e_46e4_997c_ef93cd74d222.slice/crio-c5bc8e9b6611dae1e21599c8722cb9c7a6e498f8239aa5cfc38d889d3fe845fa WatchSource:0}: Error finding container c5bc8e9b6611dae1e21599c8722cb9c7a6e498f8239aa5cfc38d889d3fe845fa: Status 404 returned error can't find the container with id c5bc8e9b6611dae1e21599c8722cb9c7a6e498f8239aa5cfc38d889d3fe845fa
Oct 30 07:11:31.099895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.099802    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:31.176318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.167317    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:31.188956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.188609    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:31.188956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.188670    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:31.509862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.509828    6738 generic.go:356] "Generic (PLEG): container finished" podID="1c1d9080-84a8-4f99-a935-17a1fefae0bb" containerID="f5a55426ed23350028d3db4c8e611f635bb1931d37a885afccb7da6abf545f50" exitCode=0
Oct 30 07:11:31.510085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.509919    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c" event={"ID":"1c1d9080-84a8-4f99-a935-17a1fefae0bb","Type":"ContainerDied","Data":"f5a55426ed23350028d3db4c8e611f635bb1931d37a885afccb7da6abf545f50"}
Oct 30 07:11:31.511771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.511737    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" event={"ID":"db64ba02-617e-46e4-997c-ef93cd74d222","Type":"ContainerStarted","Data":"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"}
Oct 30 07:11:31.511928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.511780    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" event={"ID":"db64ba02-617e-46e4-997c-ef93cd74d222","Type":"ContainerStarted","Data":"c5bc8e9b6611dae1e21599c8722cb9c7a6e498f8239aa5cfc38d889d3fe845fa"}
Oct 30 07:11:31.606088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.606036    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podStartSLOduration=1.606019337 podStartE2EDuration="1.606019337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:31.603526702 +0000 UTC m=+852.036290595" watchObservedRunningTime="2026-10-30 07:11:31.606019337 +0000 UTC m=+852.038783247"
Oct 30 07:11:31.762230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.758906    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:31.848421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.848374    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\") pod \"e1f86e2d-7f2c-477d-af3b-c60781b10626\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") "
Oct 30 07:11:31.848636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.848486    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\" (UniqueName: \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\") pod \"e1f86e2d-7f2c-477d-af3b-c60781b10626\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") "
Oct 30 07:11:31.848941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.848689    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\") pod \"e1f86e2d-7f2c-477d-af3b-c60781b10626\" (UID: \"e1f86e2d-7f2c-477d-af3b-c60781b10626\") "
Oct 30 07:11:31.849026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.849000    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca" pod "e1f86e2d-7f2c-477d-af3b-c60781b10626" (UID: "e1f86e2d-7f2c-477d-af3b-c60781b10626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:31.851098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.850736    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:31.851098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.851042    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm" pod "e1f86e2d-7f2c-477d-af3b-c60781b10626" (UID: "e1f86e2d-7f2c-477d-af3b-c60781b10626"). InnerVolumeSpecName "kube-api-access-mfshm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:31.852425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.852396    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca" pod "e1f86e2d-7f2c-477d-af3b-c60781b10626" (UID: "e1f86e2d-7f2c-477d-af3b-c60781b10626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:31.951551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.951514    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfshm\" (UniqueName: \"kubernetes.io/projected/e1f86e2d-7f2c-477d-af3b-c60781b10626-kube-api-access-mfshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:31.951551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:31.951545    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f86e2d-7f2c-477d-af3b-c60781b10626-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:32.140472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.140432    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:32.140630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.140502    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:32.433994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.433907    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:32.523184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.523155    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"
Oct 30 07:11:32.523383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.523194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b" event={"ID":"e1f86e2d-7f2c-477d-af3b-c60781b10626","Type":"ContainerDied","Data":"9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a"}
Oct 30 07:11:32.523383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.523259    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9545087fb0bd3a2c071f3edf8a41e605d28cc09e7f31e60dbe6ce369f84dcf6a"
Oct 30 07:11:32.723818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.723788    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:32.861706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.861676    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\" (UniqueName: \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\") pod \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") "
Oct 30 07:11:32.862019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.861721    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\") pod \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") "
Oct 30 07:11:32.862019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.861834    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\") pod \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\" (UID: \"1c1d9080-84a8-4f99-a935-17a1fefae0bb\") "
Oct 30 07:11:32.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.862271    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca" pod "1c1d9080-84a8-4f99-a935-17a1fefae0bb" (UID: "1c1d9080-84a8-4f99-a935-17a1fefae0bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:32.862467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.862452    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:32.864191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.864172    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca" pod "1c1d9080-84a8-4f99-a935-17a1fefae0bb" (UID: "1c1d9080-84a8-4f99-a935-17a1fefae0bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:32.865031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.865006    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm" pod "1c1d9080-84a8-4f99-a935-17a1fefae0bb" (UID: "1c1d9080-84a8-4f99-a935-17a1fefae0bb"). InnerVolumeSpecName "kube-api-access-rsclm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:32.964198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.964118    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsclm\" (UniqueName: \"kubernetes.io/projected/1c1d9080-84a8-4f99-a935-17a1fefae0bb-kube-api-access-rsclm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:32.964198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:32.964146    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1d9080-84a8-4f99-a935-17a1fefae0bb-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:33.039675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.039637    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:33.147318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.142723    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:33.147318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.142788    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:33.532281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.532251    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"
Oct 30 07:11:33.532281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.532286    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c" event={"ID":"1c1d9080-84a8-4f99-a935-17a1fefae0bb","Type":"ContainerDied","Data":"aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3"}
Oct 30 07:11:33.532762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:33.532349    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa327ef48bc617b5030cedf2dfb3b0b992111415de68f62e834d1dc26e74cfb3"
Oct 30 07:11:34.140878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:34.140843    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:34.141097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:34.140904    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:35.056177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:35.056130    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:35.146330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:35.145058    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:35.146330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:35.145127    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:36.140599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:36.140564    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 07:11:36.140977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:36.140628    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 07:11:37.045483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.045446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:37.064780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.064743    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"]
Oct 30 07:11:37.065381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065364    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e1f86e2d-7f2c-477d-af3b-c60781b10626" containerName="revoke-keys"
Oct 30 07:11:37.065381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065380    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f86e2d-7f2c-477d-af3b-c60781b10626" containerName="revoke-keys"
Oct 30 07:11:37.065530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065406    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1c1d9080-84a8-4f99-a935-17a1fefae0bb" containerName="revoke-keys"
Oct 30 07:11:37.065530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065417    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1d9080-84a8-4f99-a935-17a1fefae0bb" containerName="revoke-keys"
Oct 30 07:11:37.065530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065507    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f86e2d-7f2c-477d-af3b-c60781b10626" containerName="revoke-keys"
Oct 30 07:11:37.065530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.065520    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1d9080-84a8-4f99-a935-17a1fefae0bb" containerName="revoke-keys"
Oct 30 07:11:37.069042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.069023    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.100909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.100422    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"]
Oct 30 07:11:37.106035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.103101    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:37.106035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.103342    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:37.106035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.103538    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:37.106035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.103813    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:37.172942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.172881    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blq6k\" (UniqueName: \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.173581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.173553    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.173686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.173617    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.179344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.179314    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:37.179944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.179897    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:37.181120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.181096    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:11:37.274595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.274557    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blq6k\" (UniqueName: \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.274877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.274713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.274877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.274778    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.275764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.275735    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.278233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.278209    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.308284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.307887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blq6k\" (UniqueName: \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\") pod \"maas-api-revoke-sub-7e7ef020-fddp6\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.414363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.414331    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:37.529705 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:11:37.529668    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:11:37.724115 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:37.724083    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c62b42d_e9f6_418a_a831_f8840f148e69.slice/crio-21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231 WatchSource:0}: Error finding container 21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231: Status 404 returned error can't find the container with id 21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231
Oct 30 07:11:37.724908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:37.724875    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"]
Oct 30 07:11:38.458237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.458203    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"]
Oct 30 07:11:38.462226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.462200    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.484349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.484318    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"]
Oct 30 07:11:38.569960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.569920    6738 generic.go:356] "Generic (PLEG): container finished" podID="7c62b42d-e9f6-418a-a831-f8840f148e69" containerID="566d0b177876bc6d0d473e4f0621d5f1a720b3a22c1203d4f7850c9664a18009" exitCode=0
Oct 30 07:11:38.570152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.569966    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6" event={"ID":"7c62b42d-e9f6-418a-a831-f8840f148e69","Type":"ContainerDied","Data":"566d0b177876bc6d0d473e4f0621d5f1a720b3a22c1203d4f7850c9664a18009"}
Oct 30 07:11:38.570152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.570031    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6" event={"ID":"7c62b42d-e9f6-418a-a831-f8840f148e69","Type":"ContainerStarted","Data":"21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231"}
Oct 30 07:11:38.592637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.592600    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.592805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.592685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.592805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.592756    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plswx\" (UniqueName: \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.693934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.693890    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.694146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.694015    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.694234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.694155    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plswx\" (UniqueName: \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.694654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.694626    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.696627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.696601    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.720391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.720322    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plswx\" (UniqueName: \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\") pod \"maas-api-revoke-sub-c39a6ed4-zhs2r\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:38.844620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:38.844579    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:39.129763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.129728    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:39.152083 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:39.152048    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7712f1_8b1f_40fd_912b_166cffc6084f.slice/crio-bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee WatchSource:0}: Error finding container bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee: Status 404 returned error can't find the container with id bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee
Oct 30 07:11:39.152471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.152448    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"]
Oct 30 07:11:39.579365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.579326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r" event={"ID":"5a7712f1-8b1f-40fd-912b-166cffc6084f","Type":"ContainerStarted","Data":"d4d9ba8ff7fd0397359ed070b86260a0f90911821a975170fcf05a32774e1325"}
Oct 30 07:11:39.579832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.579391    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r" event={"ID":"5a7712f1-8b1f-40fd-912b-166cffc6084f","Type":"ContainerStarted","Data":"bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee"}
Oct 30 07:11:39.631380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.631330    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r" podStartSLOduration=1.6313123489999999 podStartE2EDuration="1.631312349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:39.629580796 +0000 UTC m=+860.062344690" watchObservedRunningTime="2026-10-30 07:11:39.631312349 +0000 UTC m=+860.064076242"
Oct 30 07:11:39.745007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.744979    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:39.814723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.814682    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\") pod \"7c62b42d-e9f6-418a-a831-f8840f148e69\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") "
Oct 30 07:11:39.814723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.814727    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\") pod \"7c62b42d-e9f6-418a-a831-f8840f148e69\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") "
Oct 30 07:11:39.814984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.814816    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\" (UniqueName: \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\") pod \"7c62b42d-e9f6-418a-a831-f8840f148e69\" (UID: \"7c62b42d-e9f6-418a-a831-f8840f148e69\") "
Oct 30 07:11:39.815119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.815095    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca" pod "7c62b42d-e9f6-418a-a831-f8840f148e69" (UID: "7c62b42d-e9f6-418a-a831-f8840f148e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:39.815379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.815353    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:39.816926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.816898    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k" pod "7c62b42d-e9f6-418a-a831-f8840f148e69" (UID: "7c62b42d-e9f6-418a-a831-f8840f148e69"). InnerVolumeSpecName "kube-api-access-blq6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:39.816926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.816915    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca" pod "7c62b42d-e9f6-418a-a831-f8840f148e69" (UID: "7c62b42d-e9f6-418a-a831-f8840f148e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:39.916541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.916451    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c62b42d-e9f6-418a-a831-f8840f148e69-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:39.916541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:39.916483    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blq6k\" (UniqueName: \"kubernetes.io/projected/7c62b42d-e9f6-418a-a831-f8840f148e69-kube-api-access-blq6k\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:40.433355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.433284    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:40.446664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.446636    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:40.587595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.587522    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"
Oct 30 07:11:40.587991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.587520    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6" event={"ID":"7c62b42d-e9f6-418a-a831-f8840f148e69","Type":"ContainerDied","Data":"21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231"}
Oct 30 07:11:40.587991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.587655    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f94a9f679975ff2746b1359022bbc97311dfd542b3c17752009c38eb662231"
Oct 30 07:11:40.589400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.589371    6738 generic.go:356] "Generic (PLEG): container finished" podID="5a7712f1-8b1f-40fd-912b-166cffc6084f" containerID="d4d9ba8ff7fd0397359ed070b86260a0f90911821a975170fcf05a32774e1325" exitCode=0
Oct 30 07:11:40.589534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.589453    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r" event={"ID":"5a7712f1-8b1f-40fd-912b-166cffc6084f","Type":"ContainerDied","Data":"d4d9ba8ff7fd0397359ed070b86260a0f90911821a975170fcf05a32774e1325"}
Oct 30 07:11:40.603736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:40.603712    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:41.005335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.005282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:41.742129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.742097    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:41.842665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.842625    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\") pod \"5a7712f1-8b1f-40fd-912b-166cffc6084f\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") "
Oct 30 07:11:41.842869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.842756    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\" (UniqueName: \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\") pod \"5a7712f1-8b1f-40fd-912b-166cffc6084f\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") "
Oct 30 07:11:41.842869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.842835    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\") pod \"5a7712f1-8b1f-40fd-912b-166cffc6084f\" (UID: \"5a7712f1-8b1f-40fd-912b-166cffc6084f\") "
Oct 30 07:11:41.843059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.843035    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca" pod "5a7712f1-8b1f-40fd-912b-166cffc6084f" (UID: "5a7712f1-8b1f-40fd-912b-166cffc6084f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:41.843448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.843424    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:41.844898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.844872    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca" pod "5a7712f1-8b1f-40fd-912b-166cffc6084f" (UID: "5a7712f1-8b1f-40fd-912b-166cffc6084f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:41.845030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.844971    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx" pod "5a7712f1-8b1f-40fd-912b-166cffc6084f" (UID: "5a7712f1-8b1f-40fd-912b-166cffc6084f"). InnerVolumeSpecName "kube-api-access-plswx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:41.944994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.944954    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plswx\" (UniqueName: \"kubernetes.io/projected/5a7712f1-8b1f-40fd-912b-166cffc6084f-kube-api-access-plswx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:41.945176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.945006    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7712f1-8b1f-40fd-912b-166cffc6084f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:41.985398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:41.985355    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:42.441725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.441689    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:11:42.442574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.442544    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5a7712f1-8b1f-40fd-912b-166cffc6084f" containerName="revoke-keys"
Oct 30 07:11:42.442787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.442764    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7712f1-8b1f-40fd-912b-166cffc6084f" containerName="revoke-keys"
Oct 30 07:11:42.442950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.442933    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7c62b42d-e9f6-418a-a831-f8840f148e69" containerName="revoke-keys"
Oct 30 07:11:42.443065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.443042    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c62b42d-e9f6-418a-a831-f8840f148e69" containerName="revoke-keys"
Oct 30 07:11:42.443283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.443254    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c62b42d-e9f6-418a-a831-f8840f148e69" containerName="revoke-keys"
Oct 30 07:11:42.443381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.443284    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7712f1-8b1f-40fd-912b-166cffc6084f" containerName="revoke-keys"
Oct 30 07:11:42.447099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.447072    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.468245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.468209    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a6a14e07-openshift-default-dockercfg-h5g7q\""
Oct 30 07:11:42.486954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.486918    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:11:42.520842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.520783    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555219    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555338    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555474    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555594    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555721    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555762    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn9d\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555836    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555913    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.560387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.555945    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677103    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677207    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677241    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677306    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677344    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677429    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677493    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677584    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.677679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.677622    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn9d\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.712418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.703714    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.712418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.704032    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.712418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.705144    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.712418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.705614    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.712418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.705634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.723318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.713067    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r" event={"ID":"5a7712f1-8b1f-40fd-912b-166cffc6084f","Type":"ContainerDied","Data":"bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee"}
Oct 30 07:11:42.723318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.713142    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd8e05e223f61581d4f1aaa5e84c7058c9b0c002fb23addfce216244689411ee"
Oct 30 07:11:42.723318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.713224    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"
Oct 30 07:11:42.724740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.724702    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn9d\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.731434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.725753    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.731434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.730436    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.742570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.742325    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\") pod \"e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") " pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:42.765915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:42.765616    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:43.014192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.014102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:43.148682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.134365    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:43.148682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.145455    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:11:43.148814 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:43.148692    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d42ae1b_497f_41d6_9129_882c6e4020ff.slice/crio-0b1601706aa50ed4d73e5ede34a6a79ee731e1a8f25d92c6dbc5b827abcca2d8 WatchSource:0}: Error finding container 0b1601706aa50ed4d73e5ede34a6a79ee731e1a8f25d92c6dbc5b827abcca2d8: Status 404 returned error can't find the container with id 0b1601706aa50ed4d73e5ede34a6a79ee731e1a8f25d92c6dbc5b827abcca2d8
Oct 30 07:11:43.151889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.151845    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:43.151973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.151939    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:43.152018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.151983    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:43.283862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.283775    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:11:43.288097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.288071    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.312515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.312482    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:11:43.400134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400100    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400177    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400270    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400372    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400392    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400421    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982rb\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400444    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400541    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.400629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.400598    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502039    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502158    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502243    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502332    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502395    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502421    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982rb\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502500    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.502685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502535    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.503264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.502841    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.503264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.503090    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.503264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.503124    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.503264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.503203    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.503605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.503418    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.505533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.505507    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.505667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.505651    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.529067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.529034    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.529219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.529134    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982rb\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.606560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.606467    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:43.723351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.723310    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerStarted","Data":"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"}
Oct 30 07:11:43.723548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.723369    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerStarted","Data":"0b1601706aa50ed4d73e5ede34a6a79ee731e1a8f25d92c6dbc5b827abcca2d8"}
Oct 30 07:11:43.766681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.766641    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:43.769130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.769087    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podStartSLOduration=1.769072955 podStartE2EDuration="1.769072955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:43.768829738 +0000 UTC m=+864.201593619" watchObservedRunningTime="2026-10-30 07:11:43.769072955 +0000 UTC m=+864.201836847"
Oct 30 07:11:43.961342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.946611    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:11:43.964343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.962563    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:43.964620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.964579    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:43.964744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:43.964654    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:11:44.733247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:44.733156    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" event={"ID":"2f007afb-cf33-42ae-9ffb-ef69e0504513","Type":"ContainerStarted","Data":"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"}
Oct 30 07:11:44.733247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:44.733240    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" event={"ID":"2f007afb-cf33-42ae-9ffb-ef69e0504513","Type":"ContainerStarted","Data":"0b6e84a479b46f7c94a15583b6c0c2040af4f86ad581a03b1d33c86d3475588a"}
Oct 30 07:11:44.771812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:44.771783    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:44.794735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:44.794684    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podStartSLOduration=1.794668801 podStartE2EDuration="1.794668801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:44.790773468 +0000 UTC m=+865.223537361" watchObservedRunningTime="2026-10-30 07:11:44.794668801 +0000 UTC m=+865.227432694"
Oct 30 07:11:44.832510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:44.832457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:45.606710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:45.606679    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:45.608260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:45.608233    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:45.608408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:45.608330    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:45.740557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:45.740485    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:45.741801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:45.741772    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:11:46.325903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:46.325865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:46.589044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:46.588946    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:46.606898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:46.606860    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:46.607199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:46.607169    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:47.627342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:47.627147    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:47.627342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:47.627215    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:47.872170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:47.860516    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:47.913766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:47.913650    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.123519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.123451    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 07:11:48.135919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.135846    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:48.415345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.409399    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.475539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.475470    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.530449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.530393    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.600366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.599446    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.620984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.620945    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:48.621316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.621263    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:48.664480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.664424    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:11:48.873443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.873395    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.875386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.874260    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:11:48.934655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.929210    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.934655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:48.929982    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:11:49.241521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.241403    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:49.486322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.421602    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:49.486322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.422349    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:11:49.487998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.487657    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.508742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.508649    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:11:49.586039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.554270    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.586039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.555008    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:11:49.587028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.586994    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:49.681347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.613576    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.681347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.614332    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:49.706323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.704468    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.706323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.704903    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:11:49.766366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.707854    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:49.766655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.766607    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:49.846320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.782670    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:49.880324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.878537    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:49.989392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:49.937371    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:11:50.268161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.268127    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:11:50.307321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.305800    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.309427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.309391    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:11:50.448500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.448455    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:11:50.514314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.514250    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.584410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.570347    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.628363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.621766    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.638320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.629063    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.639380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.639346    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:50.639600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.639572    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:50.665420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.649606    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:11:50.723037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.712468    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.794320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.782847    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v645\" (UniqueName: \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.794320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.782940    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.796215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.796184    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a6a14e07-dockercfg-xd8hq\""
Oct 30 07:11:50.797398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.797373    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a6a14e07\""
Oct 30 07:11:50.829392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.829345    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:11:50.905746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.905713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v645\" (UniqueName: \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.906184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.906161    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.908633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.907500    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:50.908800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.905066    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.943114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:50.943066    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:51.163075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.162988    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:11:51.165771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.165744    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.219971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.212800    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v645\" (UniqueName: \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:51.299799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.289574    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:51.353472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.326400    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:51.420450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.390021    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgw9\" (UniqueName: \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.448324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.445263    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.511463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.511418    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": context deadline exceeded"
Oct 30 07:11:51.520486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.520439    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:51.583061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.583027    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:11:51.585748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.585710    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgw9\" (UniqueName: \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.585916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.585774    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.586944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.586912    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.594659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.594517    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:11:51.594982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.594949    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 30 07:11:51.611857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.611825    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:11:51.614039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.614011    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:11:51.624446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.518823    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:51.635541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.625596    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:51.635541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.627917    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:51.641698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.637887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgw9\" (UniqueName: \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.767809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.757956    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:51.809579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.809549    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:11:51.809890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.809871    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:11:51.810069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.810056    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:11:51.810234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:51.810221    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:11:52.163493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.163460    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:11:52.165556 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:52.165519    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520c5be5_63b7_4335_9ecb_d74145db8677.slice/crio-c4ce67b8c58a6d619542be23c3172578ff794f04d14d320b749ed48f3e731031 WatchSource:0}: Error finding container c4ce67b8c58a6d619542be23c3172578ff794f04d14d320b749ed48f3e731031: Status 404 returned error can't find the container with id c4ce67b8c58a6d619542be23c3172578ff794f04d14d320b749ed48f3e731031
Oct 30 07:11:52.217329 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:52.214870    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595b8a9d_3a0c_485a_b26e_ea62d9714e20.slice/crio-4667354883bede0c45958cc4659a2af6b60f80194f26657fcfb53c5840dd396c WatchSource:0}: Error finding container 4667354883bede0c45958cc4659a2af6b60f80194f26657fcfb53c5840dd396c: Status 404 returned error can't find the container with id 4667354883bede0c45958cc4659a2af6b60f80194f26657fcfb53c5840dd396c
Oct 30 07:11:52.228343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.221561    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:11:52.513074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.512958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:52.611142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.610678    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:52.611142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.610736    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:52.742075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.742040    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" event={"ID":"595b8a9d-3a0c-485a-b26e-ea62d9714e20","Type":"ContainerStarted","Data":"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"}
Oct 30 07:11:52.742283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.742253    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:11:52.742381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.742288    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" event={"ID":"595b8a9d-3a0c-485a-b26e-ea62d9714e20","Type":"ContainerStarted","Data":"4667354883bede0c45958cc4659a2af6b60f80194f26657fcfb53c5840dd396c"}
Oct 30 07:11:52.744089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.744057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" event={"ID":"520c5be5-63b7-4335-9ecb-d74145db8677","Type":"ContainerStarted","Data":"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"}
Oct 30 07:11:52.744228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.744107    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" event={"ID":"520c5be5-63b7-4335-9ecb-d74145db8677","Type":"ContainerStarted","Data":"c4ce67b8c58a6d619542be23c3172578ff794f04d14d320b749ed48f3e731031"}
Oct 30 07:11:52.744228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.744143    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:11:52.800238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.800116    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" podStartSLOduration=3.800091541 podStartE2EDuration="3.800091541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:52.788582528 +0000 UTC m=+873.221346422" watchObservedRunningTime="2026-10-30 07:11:52.800091541 +0000 UTC m=+873.232855436"
Oct 30 07:11:52.834137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.834073    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" podStartSLOduration=3.834053493 podStartE2EDuration="3.834053493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:52.832426744 +0000 UTC m=+873.265190637" watchObservedRunningTime="2026-10-30 07:11:52.834053493 +0000 UTC m=+873.266817389"
Oct 30 07:11:52.924856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:52.924817    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:53.139327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.132887    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:53.606854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.606761    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:11:53.606854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.606826    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 07:11:53.731526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.731489    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"]
Oct 30 07:11:53.736413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.736383    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.759188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.759152    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"]
Oct 30 07:11:53.762576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.762545    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:11:53.762731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.762594    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:11:53.762823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.762804    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:11:53.766929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.766903    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:11:53.819867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.819833    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.820288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.820012    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnktc\" (UniqueName: \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.820288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.820053    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.921849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.921812    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnktc\" (UniqueName: \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.922027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.921862    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.922027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.921997    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.922623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.922596    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.924563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.924537    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:53.946178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:53.946143    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnktc\" (UniqueName: \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\") pod \"maas-api-revoke-sub-4f86232a-n8mmf\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:54.054162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.054121    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:54.324338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.324247    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"]
Oct 30 07:11:54.329829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.329806    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.349899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.349858    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"]
Oct 30 07:11:54.365613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.364274    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"]
Oct 30 07:11:54.380758 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:54.379432    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880509e_eef0_40ad_b522_bb1feaac449c.slice/crio-7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8 WatchSource:0}: Error finding container 7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8: Status 404 returned error can't find the container with id 7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8
Oct 30 07:11:54.428865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.428820    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.429006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.428890    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7k6\" (UniqueName: \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.429063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.429014    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.530184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.530136    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.530446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.530214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7k6\" (UniqueName: \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.530446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.530289    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.530953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.530926    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.532998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.532969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.553929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.553900    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7k6\" (UniqueName: \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\") pod \"maas-api-revoke-sub-0c1fda0a-7zcmd\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.655578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.655540    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:54.774781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.774748    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf" event={"ID":"3880509e-eef0-40ad-b522-bb1feaac449c","Type":"ContainerStarted","Data":"7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8"}
Oct 30 07:11:54.832519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.832422    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:54.996971 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:11:54.996938    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a5c09b_9f1c_4696_a281_d09163484bd5.slice/crio-573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d WatchSource:0}: Error finding container 573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d: Status 404 returned error can't find the container with id 573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d
Oct 30 07:11:54.997932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:54.997905    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"]
Oct 30 07:11:55.043570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.043533    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:55.044171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.044145    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:55.045022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.045000    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:11:55.788108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.788073    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd" event={"ID":"e7a5c09b-9f1c-4696-a281-d09163484bd5","Type":"ContainerStarted","Data":"ca5e37d8efd8793f7a5ba8565f316c248830f9439cc586acb28d71647a6408e5"}
Oct 30 07:11:55.788358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.788137    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd" event={"ID":"e7a5c09b-9f1c-4696-a281-d09163484bd5","Type":"ContainerStarted","Data":"573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d"}
Oct 30 07:11:55.790042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.790017    6738 generic.go:356] "Generic (PLEG): container finished" podID="3880509e-eef0-40ad-b522-bb1feaac449c" containerID="2d97330bb3673cb6a2bb36a1f723635648705e9435c6d1ecabd22f02acfda506" exitCode=0
Oct 30 07:11:55.790177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.790089    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf" event={"ID":"3880509e-eef0-40ad-b522-bb1feaac449c","Type":"ContainerDied","Data":"2d97330bb3673cb6a2bb36a1f723635648705e9435c6d1ecabd22f02acfda506"}
Oct 30 07:11:55.830553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:55.830494    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd" podStartSLOduration=1.830479157 podStartE2EDuration="1.830479157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:55.828369434 +0000 UTC m=+876.261133343" watchObservedRunningTime="2026-10-30 07:11:55.830479157 +0000 UTC m=+876.263243050"
Oct 30 07:11:56.731892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:56.731851    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:56.830758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:56.830725    6738 generic.go:356] "Generic (PLEG): container finished" podID="e7a5c09b-9f1c-4696-a281-d09163484bd5" containerID="ca5e37d8efd8793f7a5ba8565f316c248830f9439cc586acb28d71647a6408e5" exitCode=0
Oct 30 07:11:56.830946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:56.830816    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd" event={"ID":"e7a5c09b-9f1c-4696-a281-d09163484bd5","Type":"ContainerDied","Data":"ca5e37d8efd8793f7a5ba8565f316c248830f9439cc586acb28d71647a6408e5"}
Oct 30 07:11:57.014548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.014521    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:57.072811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.072774    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\" (UniqueName: \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\") pod \"3880509e-eef0-40ad-b522-bb1feaac449c\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") "
Oct 30 07:11:57.073025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.072868    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\") pod \"3880509e-eef0-40ad-b522-bb1feaac449c\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") "
Oct 30 07:11:57.073025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.072908    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\") pod \"3880509e-eef0-40ad-b522-bb1feaac449c\" (UID: \"3880509e-eef0-40ad-b522-bb1feaac449c\") "
Oct 30 07:11:57.073399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.073369    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca" pod "3880509e-eef0-40ad-b522-bb1feaac449c" (UID: "3880509e-eef0-40ad-b522-bb1feaac449c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:57.073663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.073640    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:57.075060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.075037    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc" pod "3880509e-eef0-40ad-b522-bb1feaac449c" (UID: "3880509e-eef0-40ad-b522-bb1feaac449c"). InnerVolumeSpecName "kube-api-access-pnktc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:57.075161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.075039    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca" pod "3880509e-eef0-40ad-b522-bb1feaac449c" (UID: "3880509e-eef0-40ad-b522-bb1feaac449c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:57.175041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.175001    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnktc\" (UniqueName: \"kubernetes.io/projected/3880509e-eef0-40ad-b522-bb1feaac449c-kube-api-access-pnktc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:57.175041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.175036    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880509e-eef0-40ad-b522-bb1feaac449c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:57.839823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.839793    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"
Oct 30 07:11:57.840250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.839831    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf" event={"ID":"3880509e-eef0-40ad-b522-bb1feaac449c","Type":"ContainerDied","Data":"7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8"}
Oct 30 07:11:57.840250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.839885    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b40e92a43ae32a1af10946fb19626c2e62080cc0ed3afd960a32fd6e8b139e8"
Oct 30 07:11:57.998686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:57.998657    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:58.085338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.085287    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\") pod \"e7a5c09b-9f1c-4696-a281-d09163484bd5\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") "
Oct 30 07:11:58.085527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.085407    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\") pod \"e7a5c09b-9f1c-4696-a281-d09163484bd5\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") "
Oct 30 07:11:58.085609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.085591    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\" (UniqueName: \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\") pod \"e7a5c09b-9f1c-4696-a281-d09163484bd5\" (UID: \"e7a5c09b-9f1c-4696-a281-d09163484bd5\") "
Oct 30 07:11:58.085770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.085745    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca" pod "e7a5c09b-9f1c-4696-a281-d09163484bd5" (UID: "e7a5c09b-9f1c-4696-a281-d09163484bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:11:58.086141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.086111    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:58.087638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.087617    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6" pod "e7a5c09b-9f1c-4696-a281-d09163484bd5" (UID: "e7a5c09b-9f1c-4696-a281-d09163484bd5"). InnerVolumeSpecName "kube-api-access-sn7k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:11:58.087729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.087641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca" pod "e7a5c09b-9f1c-4696-a281-d09163484bd5" (UID: "e7a5c09b-9f1c-4696-a281-d09163484bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:11:58.187688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.187651    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn7k6\" (UniqueName: \"kubernetes.io/projected/e7a5c09b-9f1c-4696-a281-d09163484bd5-kube-api-access-sn7k6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:58.187688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.187683    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a5c09b-9f1c-4696-a281-d09163484bd5-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:11:58.664214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.664175    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:11:58.849120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.849086    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"
Oct 30 07:11:58.849120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.849114    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd" event={"ID":"e7a5c09b-9f1c-4696-a281-d09163484bd5","Type":"ContainerDied","Data":"573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d"}
Oct 30 07:11:58.849607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:11:58.849165    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="573913020fc0dc7241fd89a2aaa05684e1bf332c5194ae335bdca848974e932d"
Oct 30 07:12:00.525915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:00.525879    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:02.407884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:02.407847    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:02.522828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:02.522785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:02.652307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:02.649271    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:03.761031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:03.760995    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:12:03.761566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:03.761485    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:12:03.863603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:03.863563    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:04.188020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:04.187980    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:04.541954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:04.541871    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:04.696316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:04.695641    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:05.045723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:05.045687    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:05.218315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:05.214707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:07.419317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.407434    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:07.519910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.519180    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:07.635029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.613384    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:07.644068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.643413    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:07.725502 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:12:07.713190    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:12:07.862320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.861663    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:07.887314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.881375    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:12:07.918318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:07.917859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:08.103762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.092558    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:12:08.103762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.097510    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" containerID="cri-o://5829f826d0fcff4c8d44ac1d40d7216566b961dd3d387ae042f4e48b782651c6" gracePeriod=30
Oct 30 07:12:08.121487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.121435    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 07:12:08.298316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.282396    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.433826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.431395    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.433826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.432277    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:08.530317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.529459    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.534325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.530546    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:12:08.653341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.622410    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:12:08.653341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.622858    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:12:08.695503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.654424    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.695503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.654864    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:12:08.904319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.882607    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.904319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.883279    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:12:08.904319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.884609    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.904319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.884887    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.904319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:08.885714    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:12:09.294346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.292760    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.294346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.293453    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:12:09.454322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.431808    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.563931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.563787    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.605816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.605779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:09.640018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.639972    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.664876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.663472    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.757134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.757100    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:09.900740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.888381    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.901142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.901097    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.906152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.906120    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:12:09.950403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.948394    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.960461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:09.960418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:10.303378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.295183    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.432475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.432430    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.590517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.579779    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.684321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.659430    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.684911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.684261    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.990952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.990897    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.991583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:10.991549    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:11.015974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.015920    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.251562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.251468    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:11.251836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.251801    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.265380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.265332    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.407437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.407384    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:12:11.465846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.465388    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": context deadline exceeded"
Oct 30 07:12:11.473553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.466129    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.617026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.616920    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.714319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.619179    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.714319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.693704    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.714319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.693750    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:12:12.018497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:11.991434    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.019795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.018879    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.021336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.021280    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.411425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.411377    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.480456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.480401    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.617637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.617595    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:12:12.735378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.722929    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.749323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.731373    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:12.791172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.736192    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:12.998910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:12.998810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:13.056861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.021876    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:13.056861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.022678    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:12:13.056861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.035823    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:13.139317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.126988    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:12:13.170330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.160031    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e7a5c09b-9f1c-4696-a281-d09163484bd5" containerName="revoke-keys"
Oct 30 07:12:13.217648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.217609    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a5c09b-9f1c-4696-a281-d09163484bd5" containerName="revoke-keys"
Oct 30 07:12:13.217964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.217949    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3880509e-eef0-40ad-b522-bb1feaac449c" containerName="revoke-keys"
Oct 30 07:12:13.218053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.218042    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3880509e-eef0-40ad-b522-bb1feaac449c" containerName="revoke-keys"
Oct 30 07:12:13.218393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.218376    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a5c09b-9f1c-4696-a281-d09163484bd5" containerName="revoke-keys"
Oct 30 07:12:13.242785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.242756    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3880509e-eef0-40ad-b522-bb1feaac449c" containerName="revoke-keys"
Oct 30 07:12:13.319563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.315633    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.418320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.394525    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.451627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.451581    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hms\" (UniqueName: \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.461266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.412988    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:13.461266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.460157    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.546831    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a163b2-dockercfg-kr2lb\""
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.546967    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a163b2\""
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.550415    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.557714    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.561174    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hms\" (UniqueName: \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.561389    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.566344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.562441    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.590352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.590256    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:12:13.598008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.597941    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hms\" (UniqueName: \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\") pod \"payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.754741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.748931    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:12:13.782133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.782103    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:12:13.787086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.786755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:13.794451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.794411    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:13.918973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.895917    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:13.920016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:13.919803    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:14.074416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.074365    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 07:12:14.088752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.088719    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:12:14.107328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.107133    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:12:14.181130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.180661    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:14.185787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.183908    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:12:14.196684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.196658    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:12:14.214926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.214897    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:12:14.215147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.215129    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:12:14.218643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.217706    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.223437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.223133    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:12:14.223437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.223249    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385008    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385089    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385120    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385172    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjldv\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385262    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385307    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385333    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385399    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385432    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385458    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385482    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385537    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9j6\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385573    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385681    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.385782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385741    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.386370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385822    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.386370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385849    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.386370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.385921    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.454952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.454919    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:12:14.455587 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:14.455559    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be70749_a046_4fd3_9f79_80069fc179e7.slice/crio-7132af4f46c7b696cbc93772a9e2ac5b4c0cec8ab9ec2fa07abd91d3d9e5a495 WatchSource:0}: Error finding container 7132af4f46c7b696cbc93772a9e2ac5b4c0cec8ab9ec2fa07abd91d3d9e5a495: Status 404 returned error can't find the container with id 7132af4f46c7b696cbc93772a9e2ac5b4c0cec8ab9ec2fa07abd91d3d9e5a495
Oct 30 07:12:14.496815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.496775    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.496995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.496844    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.496995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.496900    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.496995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.496924    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.496995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.496971    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.497165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497032    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497083    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497106    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497159    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjldv\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497249    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497276    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.497389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497313    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497374    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.497491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497404    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497429    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.497491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497454    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.497590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497505    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9j6\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.497590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.497544    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.498040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.498016    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.498468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.498440    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499171    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499228    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499661    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499723    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499801    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.500206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.499884    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.501194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.501142    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.501444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.501406    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.501547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.501463    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.501763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.501713    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.502573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.502545    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.543428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.543399    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.543428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.543413    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.543605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.543516    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9j6\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.547607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.547585    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjldv\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.549506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.549484    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:14.558932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.558910    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:14.880366 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:14.880322    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae54530_42b1_4a0a_bd80_4b53f5038a51.slice/crio-cb9c4120d105dacb2be070da097f7f6cc4d85ea8c5334df6ae3196be31e4876f WatchSource:0}: Error finding container cb9c4120d105dacb2be070da097f7f6cc4d85ea8c5334df6ae3196be31e4876f: Status 404 returned error can't find the container with id cb9c4120d105dacb2be070da097f7f6cc4d85ea8c5334df6ae3196be31e4876f
Oct 30 07:12:14.881197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.881165    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:12:14.881810 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:14.881785    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e34096d_9005_409a_b595_010c2ea73a79.slice/crio-ae47d7f5e3e16ccc44138b82a0534b642b5113b5c63643618203b2d1c01de950 WatchSource:0}: Error finding container ae47d7f5e3e16ccc44138b82a0534b642b5113b5c63643618203b2d1c01de950: Status 404 returned error can't find the container with id ae47d7f5e3e16ccc44138b82a0534b642b5113b5c63643618203b2d1c01de950
Oct 30 07:12:14.883588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.883543    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.883696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.883640    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.883696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.883686    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.884846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.884801    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.884926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.884891    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.884968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.884935    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:14.885832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.885808    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:12:14.918147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.918118    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"]
Oct 30 07:12:14.925020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.924996    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:14.938630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.938599    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"]
Oct 30 07:12:14.940392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.940367    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:12:14.940590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.940565    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:12:14.940726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.940597    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:12:14.940873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:14.940839    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:12:15.005982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.005947    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.006182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.005988    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.006182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.006162    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mx7\" (UniqueName: \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.108250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.108220    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mx7\" (UniqueName: \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.108502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.108483    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.108562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.108523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.109419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.109400    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.110879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.110857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.132959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.132884    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mx7\" (UniqueName: \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\") pod \"maas-api-revoke-sub-cd706324-k7bng\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.206239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.206191    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" event={"ID":"7e34096d-9005-409a-b595-010c2ea73a79","Type":"ContainerStarted","Data":"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"}
Oct 30 07:12:15.206441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.206257    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" event={"ID":"7e34096d-9005-409a-b595-010c2ea73a79","Type":"ContainerStarted","Data":"ae47d7f5e3e16ccc44138b82a0534b642b5113b5c63643618203b2d1c01de950"}
Oct 30 07:12:15.208455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.208426    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" event={"ID":"9be70749-a046-4fd3-9f79-80069fc179e7","Type":"ContainerStarted","Data":"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"}
Oct 30 07:12:15.208598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.208483    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:15.208598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.208505    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" event={"ID":"9be70749-a046-4fd3-9f79-80069fc179e7","Type":"ContainerStarted","Data":"7132af4f46c7b696cbc93772a9e2ac5b4c0cec8ab9ec2fa07abd91d3d9e5a495"}
Oct 30 07:12:15.210719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.210694    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" event={"ID":"eae54530-42b1-4a0a-bd80-4b53f5038a51","Type":"ContainerStarted","Data":"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"}
Oct 30 07:12:15.210820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.210741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" event={"ID":"eae54530-42b1-4a0a-bd80-4b53f5038a51","Type":"ContainerStarted","Data":"cb9c4120d105dacb2be070da097f7f6cc4d85ea8c5334df6ae3196be31e4876f"}
Oct 30 07:12:15.256160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.256105    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podStartSLOduration=1.2560894280000001 podStartE2EDuration="1.256089428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.253420446 +0000 UTC m=+895.686184339" watchObservedRunningTime="2026-10-30 07:12:15.256089428 +0000 UTC m=+895.688853322"
Oct 30 07:12:15.299963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.299895    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podStartSLOduration=2.29987526 podStartE2EDuration="2.29987526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.29902344 +0000 UTC m=+895.731787350" watchObservedRunningTime="2026-10-30 07:12:15.29987526 +0000 UTC m=+895.732639153"
Oct 30 07:12:15.344632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.344552    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" podStartSLOduration=3.344531855 podStartE2EDuration="3.344531855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.340916517 +0000 UTC m=+895.773680409" watchObservedRunningTime="2026-10-30 07:12:15.344531855 +0000 UTC m=+895.777295750"
Oct 30 07:12:15.374379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.374346    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:15.552009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.550469    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:15.577838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.577757    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:15.787760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:15.783911    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"]
Oct 30 07:12:15.828952 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:15.828748    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4816a68_f98c_46ab_9a66_3a803f868e82.slice/crio-c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9 WatchSource:0}: Error finding container c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9: Status 404 returned error can't find the container with id c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9
Oct 30 07:12:16.229378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.229341    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"]
Oct 30 07:12:16.247111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.247065    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:16.251154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.251124    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng" event={"ID":"f4816a68-f98c-46ab-9a66-3a803f868e82","Type":"ContainerStarted","Data":"c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9"}
Oct 30 07:12:16.251340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.251240    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.251461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.251434    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"]
Oct 30 07:12:16.420679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.420642    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rhz\" (UniqueName: \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.420959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.420938    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.421074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.421058    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.522808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.522641    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rhz\" (UniqueName: \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.522808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.522713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.522808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.522745    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.552984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.523555    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.552984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.525934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.552984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.547234    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rhz\" (UniqueName: \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\") pod \"maas-api-revoke-sub-7738eaca-zb74b\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:16.552984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.551157    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:16.552984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.551202    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:16.560669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.560644    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:16.560804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.560686    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:16.600258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:16.599970    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:17.022163 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:17.022114    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4c0d6c_f15a_4651_be89_cd1235d5943a.slice/crio-e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8 WatchSource:0}: Error finding container e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8: Status 404 returned error can't find the container with id e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8
Oct 30 07:12:17.023180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.023104    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"]
Oct 30 07:12:17.256882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.256781    6738 generic.go:356] "Generic (PLEG): container finished" podID="f4816a68-f98c-46ab-9a66-3a803f868e82" containerID="f3090bc07c323fbc35ffd2a498d9d8e059fcfbbaa9d50c7d58d3005585dcbc49" exitCode=0
Oct 30 07:12:17.256882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.256863    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng" event={"ID":"f4816a68-f98c-46ab-9a66-3a803f868e82","Type":"ContainerDied","Data":"f3090bc07c323fbc35ffd2a498d9d8e059fcfbbaa9d50c7d58d3005585dcbc49"}
Oct 30 07:12:17.258931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.258911    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b" event={"ID":"da4c0d6c-f15a-4651-be89-cd1235d5943a","Type":"ContainerStarted","Data":"e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8"}
Oct 30 07:12:17.550404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.550286    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:17.550404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.550373    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:17.559937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.559909    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:17.560066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:17.559956    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:18.291332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.272972    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b" event={"ID":"da4c0d6c-f15a-4651-be89-cd1235d5943a","Type":"ContainerStarted","Data":"36126248fd89926333ee00c86017de7b63ad6b15751bd78c3a0b521440e5fbbc"}
Oct 30 07:12:18.337313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.336094    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b" podStartSLOduration=2.336070582 podStartE2EDuration="2.336070582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:18.329085467 +0000 UTC m=+898.761849388" watchObservedRunningTime="2026-10-30 07:12:18.336070582 +0000 UTC m=+898.768834475"
Oct 30 07:12:18.433318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.429621    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:18.601478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.599926    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:18.601478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.599986    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:18.601982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.601938    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:18.602151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.602128    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:18.740053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.740029    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:18.832758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.832718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\") pod \"f4816a68-f98c-46ab-9a66-3a803f868e82\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") "
Oct 30 07:12:18.832978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.832799    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\" (UniqueName: \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\") pod \"f4816a68-f98c-46ab-9a66-3a803f868e82\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") "
Oct 30 07:12:18.832978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.832854    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\") pod \"f4816a68-f98c-46ab-9a66-3a803f868e82\" (UID: \"f4816a68-f98c-46ab-9a66-3a803f868e82\") "
Oct 30 07:12:18.833921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.833896    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca" pod "f4816a68-f98c-46ab-9a66-3a803f868e82" (UID: "f4816a68-f98c-46ab-9a66-3a803f868e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:12:18.836152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.836121    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca" pod "f4816a68-f98c-46ab-9a66-3a803f868e82" (UID: "f4816a68-f98c-46ab-9a66-3a803f868e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:12:18.836268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.836152    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7" pod "f4816a68-f98c-46ab-9a66-3a803f868e82" (UID: "f4816a68-f98c-46ab-9a66-3a803f868e82"). InnerVolumeSpecName "kube-api-access-p8mx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:12:18.935055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.935017    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:18.935055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.935051    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8mx7\" (UniqueName: \"kubernetes.io/projected/f4816a68-f98c-46ab-9a66-3a803f868e82-kube-api-access-p8mx7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:18.935055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:18.935063    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4816a68-f98c-46ab-9a66-3a803f868e82-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.200372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.199377    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8srb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:12:19.200372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.199445    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:12:19.325252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.325211    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng" event={"ID":"f4816a68-f98c-46ab-9a66-3a803f868e82","Type":"ContainerDied","Data":"c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9"}
Oct 30 07:12:19.325722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.325283    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a98a51a62adb16430c6d769c3202007e9bf8abad4fb6ec1914f47ec9b5d4f9"
Oct 30 07:12:19.325722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.325390    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"
Oct 30 07:12:19.327862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.327832    6738 generic.go:356] "Generic (PLEG): container finished" podID="da4c0d6c-f15a-4651-be89-cd1235d5943a" containerID="36126248fd89926333ee00c86017de7b63ad6b15751bd78c3a0b521440e5fbbc" exitCode=0
Oct 30 07:12:19.328139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.327883    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b" event={"ID":"da4c0d6c-f15a-4651-be89-cd1235d5943a","Type":"ContainerDied","Data":"36126248fd89926333ee00c86017de7b63ad6b15751bd78c3a0b521440e5fbbc"}
Oct 30 07:12:19.551634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.550864    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:19.551634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.550927    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:19.559954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.559917    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:19.560146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.559980    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:19.744026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.743992    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:12:19.849021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.848940    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849242    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:12:19.849403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849378    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849449    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849563    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849658    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849700    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849717    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849810    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.849855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.849847    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\") pod \"209e1900-beb5-4db6-b6ef-93f51023684e\" (UID: \"209e1900-beb5-4db6-b6ef-93f51023684e\") "
Oct 30 07:12:19.850099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.850071    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:12:19.850358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.850325    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:12:19.850460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.850410    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209e1900-beb5-4db6-b6ef-93f51023684e-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.850460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.850429    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.851286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.851259    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:12:19.851592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.851563    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:12:19.853144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.852741    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "kube-api-access-jt8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:12:19.853645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.853623    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:12:19.854015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.853982    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:12:19.854394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.854372    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token" pod "209e1900-beb5-4db6-b6ef-93f51023684e" (UID: "209e1900-beb5-4db6-b6ef-93f51023684e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:12:19.946095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.946065    6738 scope.go:122] "RemoveContainer" containerID="e48262df1251bdab613c235b13de6c62163ac407878e72a9034cac458c2bb41d"
Oct 30 07:12:19.951556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951527    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951735    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8fr\" (UniqueName: \"kubernetes.io/projected/209e1900-beb5-4db6-b6ef-93f51023684e-kube-api-access-jt8fr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951761    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951780    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209e1900-beb5-4db6-b6ef-93f51023684e-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951794    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951808    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.951843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.951823    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209e1900-beb5-4db6-b6ef-93f51023684e-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:19.959051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:19.958984    6738 scope.go:122] "RemoveContainer" containerID="5829f826d0fcff4c8d44ac1d40d7216566b961dd3d387ae042f4e48b782651c6"
Oct 30 07:12:20.023786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.023751    6738 scope.go:122] "RemoveContainer" containerID="46353d5fff893ee79f832d6ed459af639410957108ab8d46ec3e54dbe1c4df4c"
Oct 30 07:12:20.039889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.039868    6738 scope.go:122] "RemoveContainer" containerID="9f0700353067860a6f08077278995f5c84c3be1e6f94f24cc05578ea2546940d"
Oct 30 07:12:20.336312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.336170    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" event={"ID":"209e1900-beb5-4db6-b6ef-93f51023684e","Type":"ContainerDied","Data":"5829f826d0fcff4c8d44ac1d40d7216566b961dd3d387ae042f4e48b782651c6"}
Oct 30 07:12:20.336312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.336243    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4" event={"ID":"209e1900-beb5-4db6-b6ef-93f51023684e","Type":"ContainerDied","Data":"b17d90d614a36c72a89ba81a5f8d6e08cf63f4df6bd2ba76d224533c311531ef"}
Oct 30 07:12:20.336312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.336199    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"
Oct 30 07:12:20.442370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.442312    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:12:20.446325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.446271    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8srb4"]
Oct 30 07:12:20.552621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.552386    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:20.552621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.552468    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:20.560480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.560441    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:20.560670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.560557    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:20.630078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.630045    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:20.635566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.635506    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:20.842830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.842707    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\" (UniqueName: \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\") pod \"da4c0d6c-f15a-4651-be89-cd1235d5943a\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") "
Oct 30 07:12:20.845154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.845115    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\") pod \"da4c0d6c-f15a-4651-be89-cd1235d5943a\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") "
Oct 30 07:12:20.845614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.845591    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\") pod \"da4c0d6c-f15a-4651-be89-cd1235d5943a\" (UID: \"da4c0d6c-f15a-4651-be89-cd1235d5943a\") "
Oct 30 07:12:20.845849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.845599    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca" pod "da4c0d6c-f15a-4651-be89-cd1235d5943a" (UID: "da4c0d6c-f15a-4651-be89-cd1235d5943a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:12:20.845924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.845832    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz" pod "da4c0d6c-f15a-4651-be89-cd1235d5943a" (UID: "da4c0d6c-f15a-4651-be89-cd1235d5943a"). InnerVolumeSpecName "kube-api-access-d7rhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:12:20.846473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.846453    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7rhz\" (UniqueName: \"kubernetes.io/projected/da4c0d6c-f15a-4651-be89-cd1235d5943a-kube-api-access-d7rhz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:20.846589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.846576    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:20.850224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.850195    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca" pod "da4c0d6c-f15a-4651-be89-cd1235d5943a" (UID: "da4c0d6c-f15a-4651-be89-cd1235d5943a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.938936    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939840    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da4c0d6c-f15a-4651-be89-cd1235d5943a" containerName="revoke-keys"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939861    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4c0d6c-f15a-4651-be89-cd1235d5943a" containerName="revoke-keys"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939889    6738 cpu_manager.go:414] "containerMap: removing container" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939896    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939921    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f4816a68-f98c-46ab-9a66-3a803f868e82" containerName="revoke-keys"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.939928    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4816a68-f98c-46ab-9a66-3a803f868e82" containerName="revoke-keys"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.940065    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4c0d6c-f15a-4651-be89-cd1235d5943a" containerName="revoke-keys"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.940079    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" containerName="istio-proxy"
Oct 30 07:12:20.941057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.940091    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4816a68-f98c-46ab-9a66-3a803f868e82" containerName="revoke-keys"
Oct 30 07:12:20.956520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.955091    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c0d6c-f15a-4651-be89-cd1235d5943a-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:12:20.960501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.959495    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:12:20.964536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:20.964507    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.029952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.028551    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:12:21.029952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.028706    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.035617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.035587    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:12:21.157801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.157742    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.157984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.157866    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.157984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.157910    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.157984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.157960    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158078    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158095    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssszf\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158233    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158255    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158288    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158376    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158469    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5w6\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158548    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158577    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158661    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158698    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.158809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158724    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158746    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.158809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.158786    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.259819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.259767    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.259819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.259828    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.259856    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssszf\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.259936    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.259963    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260004    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260033    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260061    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5w6\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260098    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260123    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260176    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260202    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260226    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260251    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260279    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260287    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260472    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260514    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.260861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260558    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260658    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.260861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260686    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.261129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260953    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.261129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.260967    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.261129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.261025    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.261288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.261250    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.261377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.261339    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.261431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.261408    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.261622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.261601    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.263237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.263215    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.263508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.263489    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.263637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.263617    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.263719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.263703    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.311504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.311428    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssszf\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.320734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.318932    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.320734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.320662    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5w6\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.321711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.321678    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.338011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.337982    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:21.345304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.345269    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"
Oct 30 07:12:21.345462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.345427    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b" event={"ID":"da4c0d6c-f15a-4651-be89-cd1235d5943a","Type":"ContainerDied","Data":"e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8"}
Oct 30 07:12:21.345504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.345494    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f320b92b5c81d66d08fe4c2eb52e2f725eab2b4c38801dae489f9b719c5ad8"
Oct 30 07:12:21.350250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.350225    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:21.654329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.647822    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:21.654329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.647880    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:21.654329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.648318    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:21.654329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.648359    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:21.783366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.780530    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209e1900-beb5-4db6-b6ef-93f51023684e" path="/var/lib/kubelet/pods/209e1900-beb5-4db6-b6ef-93f51023684e/volumes"
Oct 30 07:12:21.783710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.783683    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:12:21.794277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.794230    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:21.794464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.794343    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:21.794464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.794398    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:21.949833 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:21.947730    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b217882_7f4a_4546_aaff_79d93995baf6.slice/crio-f1bdbb7ab93d61e5c8c00aec95acb1a8691c402217b07daa52507942ffb01d04 WatchSource:0}: Error finding container f1bdbb7ab93d61e5c8c00aec95acb1a8691c402217b07daa52507942ffb01d04: Status 404 returned error can't find the container with id f1bdbb7ab93d61e5c8c00aec95acb1a8691c402217b07daa52507942ffb01d04
Oct 30 07:12:21.953335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.951431    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:12:21.953573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.953416    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:21.953573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.953499    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:21.953573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:21.953544    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:22.363355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.363246    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" event={"ID":"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c","Type":"ContainerStarted","Data":"b621dbc3bd3d10727a7cd8ff3fc5a0d2733e4677d290e34f0f1988be2cf97556"}
Oct 30 07:12:22.363355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.363321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" event={"ID":"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c","Type":"ContainerStarted","Data":"b6f0553008e922a94b4a61b23bc66df4a385ae5d9a4a30fa069c8992d8741d9a"}
Oct 30 07:12:22.371736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.370903    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" event={"ID":"1b217882-7f4a-4546-aaff-79d93995baf6","Type":"ContainerStarted","Data":"9589bd1e37c172faa721f5a0301b96a8c34a21c33552aaa88df4e6ed19462baf"}
Oct 30 07:12:22.371736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.370964    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" event={"ID":"1b217882-7f4a-4546-aaff-79d93995baf6","Type":"ContainerStarted","Data":"f1bdbb7ab93d61e5c8c00aec95acb1a8691c402217b07daa52507942ffb01d04"}
Oct 30 07:12:22.496482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.496407    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podStartSLOduration=2.496384326 podStartE2EDuration="2.496384326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:22.421951416 +0000 UTC m=+902.854715310" watchObservedRunningTime="2026-10-30 07:12:22.496384326 +0000 UTC m=+902.929148220"
Oct 30 07:12:22.516417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.498780    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podStartSLOduration=2.498763315 podStartE2EDuration="2.498763315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:22.467997379 +0000 UTC m=+902.900761300" watchObservedRunningTime="2026-10-30 07:12:22.498763315 +0000 UTC m=+902.931527210"
Oct 30 07:12:22.555346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.555040    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 07:12:22.555346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.555097    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 07:12:22.560432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.560379    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 07:12:22.560432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:22.560435    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 07:12:23.191165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.191121    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:23.401350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.375840    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:23.401350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.378223    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:23.442377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.427187    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:23.507360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.488900    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:23.507690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.507651    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:23.508274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.508244    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:23.766380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.751411    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:23.951320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.948429    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Oct 30 07:12:23.986437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:23.986388    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.053504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.053393    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.054487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.054455    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.063319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.059512    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.339834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.339741    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:24.340073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.340044    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:24.353778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.353737    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:24.354054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.354023    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:24.425772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.424615    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.470505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.470448    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.641330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.640848    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:24.641330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.641179    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:24.641330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.641215    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.645336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.644381    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.717494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.717455    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:24.717757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.717729    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:24.761354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.753410    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:12:24.761354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:24.753849    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:12:25.009504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.009400    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.059811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.059764    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:25.083319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.060403    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:12:25.084904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.084870    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:25.113319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.088738    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:12:25.114947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.114906    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.115593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.115570    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:12:25.291378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.289719    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:25.291378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.291049    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:25.357348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.324068    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:12:25.373367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.341808    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:25.394465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.394419    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:25.427321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.374100    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:25.445439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.445386    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:25.457405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.426405    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.470321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.462746    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:12:25.546376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.543803    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.546376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.544204    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:25.677150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.677107    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:25.695441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.680453    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.695441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.685086    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:12:25.695441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.685178    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:12:25.766484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.734398    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:25.766484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.734469    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.766484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.752704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:25.814867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:25.814764    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.064319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.062966    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:26.064319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.064252    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:26.080383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.065042    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.106376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.091936    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:26.106376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.093129    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:26.106376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.093179    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.132320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.108980    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:12:26.132606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.118383    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.334732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.334632    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:12:26.357380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.347387    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:26.357380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.347449    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:26.417386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.414434    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:26.417386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.414496    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:26.475894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.475837    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.559345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.557326    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.716396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.684633    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.745342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.698759    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.869348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.864397    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.956320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:26.953969    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:27.195433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.129435    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.195433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.133087    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.227372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.197362    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.387330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.387147    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:27.387330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.387232    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:27.390331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.388028    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:27.390331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.388119    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:27.463994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.463891    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:27.500567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.500512    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.588396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.559404    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.734329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.734206    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:27.782448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.745183    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.788363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.761673    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:27.877687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:27.877642    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.000450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.000345    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:12:28.029579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.029543    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:12:28.029969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.029902    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.052326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.049855    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3bda71-openshift-default-dockercfg-nl5tk\""
Oct 30 07:12:28.126178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.126138    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.126491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.126466    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.126638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.126622    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.126775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.126761    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.127120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.127041    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.127314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.127277    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.127527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.127454    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzc5\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.127725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.127708    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.127889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.127871    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.136469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.136424    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.137091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.137059    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:28.137526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.137501    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.223441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.223387    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.232233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.232189    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.233046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.233021    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.233592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.233574    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzc5\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.234153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.234132    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.235086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.235067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.235563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.235543    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.235993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.235976    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.241137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.241111    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.302522    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.235906    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.234955    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.232919    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.295616    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.301550    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzc5\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.235456    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.308419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.233492    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.337036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.336999    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.350719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.350682    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:28.351016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.350988    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:28.376320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.359677    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:28.376320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.359742    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:28.376320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.361679    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\") pod \"e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") " pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.406321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.402534    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:28.431326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.430617    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:12:28.451332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.441989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:28.546874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.546830    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:12:28.589262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.561508    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.818334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.803515    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:28.818334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.803937    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:12:28.842396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:28.842037    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:29.202319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.145026    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:29.233454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.153517    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:12:29.262319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.261464    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:29.321319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.320734    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:12:29.350319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.347096    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:29.350319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.347154    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:29.375800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.375655    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:29.375800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.375714    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:29.595325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.589468    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:29.610363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.604483    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.766752    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.766825    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl46t\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.766862    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.766942    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.766988    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.767168    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.767256    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.767339    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.768316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.767388    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.814398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.813854    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:29.814398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.814247    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:29.817163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.817119    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:29.817445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.817408    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:29.817650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.817610    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:29.871275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.871215    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.871602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.871576    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.871857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.871839    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.871989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.871973    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl46t\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.872098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.872084    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.872256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.872241    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.872399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.872381    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.872673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.872655    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.872844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.872829    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.873537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.873516    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.903461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.899956    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.903685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.900284    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.903898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.900593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.904026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.903094    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.914987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.914952    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.947458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.947401    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:12:29.961465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.961429    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl46t\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.987333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.979162    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.987333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.979812    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:29.987904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.987874    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:12:30.009772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:29.989398    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:30.144331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.143414    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:30.144331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.144259    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:12:30.144571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.144403    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:12:30.163336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.160523    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:30.296254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.230163    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:30.454616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.424680    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:30.588525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.588475    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:30.593866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.508508    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:30.615821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.464107    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:30.616421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:30.616384    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:31.065726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.065633    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:12:31.098842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.098808    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:12:31.121410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.100089    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:31.208322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.182418    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:31.378582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.378527    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:31.388672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.388625    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:31.437553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.427165    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:31.580792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.559767    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:31.580792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.559833    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:31.602001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.601951    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:31.796509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.796409    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.090757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:31.925202    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:32.128474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.092084    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.246315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.124980    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:32.246742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.125163    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.247069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.142762    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.263549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.142841    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.367414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.202550    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.475319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.462480    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.633201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.632508    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:32.642520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.642476    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:32.642644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.642543    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:32.642644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.642623    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:32.642735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.642646    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:32.648464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.648371    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:32.655326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.649467    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:32.655326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.649560    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:32.655326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.649605    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:32.738463 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:12:32.738421    6738 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s"
Oct 30 07:12:32.738626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738486    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerStarted","Data":"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"}
Oct 30 07:12:32.738626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738543    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerStarted","Data":"b020bfec877e95f1fb17ea952def4c8aaac35ee83fb4188dcebc7b981992c724"}
Oct 30 07:12:32.738626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738578    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:12:32.738846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738649    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:32.738927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738905    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:12:32.738982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738956    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:12:32.739034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.738989    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:12:32.740146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.740124    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:12:32.888863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.888740    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:32.923764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:32.923699    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podStartSLOduration=5.923679561 podStartE2EDuration="5.923679561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:32.865673341 +0000 UTC m=+913.298437236" watchObservedRunningTime="2026-10-30 07:12:32.923679561 +0000 UTC m=+913.356443455"
Oct 30 07:12:33.356317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.354263    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:12:33.356317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.354318    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:12:33.356317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.354344    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 07:12:33.356317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.354340    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 07:12:33.405936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.405740    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:33.462580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.459810    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:33.746706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.746660    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" event={"ID":"21e07602-bf26-4e36-a477-eb9c9555931a","Type":"ContainerStarted","Data":"96fb11b269c6d267e1499d787cda5a547ab2660bf51584e3d80ed982cf1cfc29"}
Oct 30 07:12:33.746911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.746734    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" event={"ID":"21e07602-bf26-4e36-a477-eb9c9555931a","Type":"ContainerStarted","Data":"84c1c7fb936a1f09100e24e712529d7823f396c3b9759cf49ff75861013c71d5"}
Oct 30 07:12:33.748107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.748080    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:33.774342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.774245    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:12:33.811832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:33.811760    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podStartSLOduration=4.811742039 podStartE2EDuration="4.811742039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:33.807145741 +0000 UTC m=+914.239909636" watchObservedRunningTime="2026-10-30 07:12:33.811742039 +0000 UTC m=+914.244505936"
Oct 30 07:12:34.152340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.150372    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:34.152340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.151706    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:34.152340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.151759    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:34.621043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.620935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:34.854733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.854689    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:34.856284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.856263    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:34.857367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.857059    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:12:34.991513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:34.991477    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:35.020939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.020911    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:35.025750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.025706    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:12:35.147042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.146993    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:35.147227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.147057    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:35.688322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.685941    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:35.827505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:35.827466    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:36.144275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:36.144231    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:36.144471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:36.144313    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:36.897070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:36.897030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:37.144068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.144030    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:37.144232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.144089    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:37.532396 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:12:37.532355    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:12:37.601645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.601606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:37.729263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.729213    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:37.749486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.749131    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:12:37.752983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.752954    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:37.798592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.798498    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:12:37.799802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.799767    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3bda71\""
Oct 30 07:12:37.812458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.812410    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3bda71-dockercfg-7cwcd\""
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.959239    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqmq\" (UniqueName: \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:37.959330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.104546    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqmq\" (UniqueName: \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.109396    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.110687    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.148702    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqmq\" (UniqueName: \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\") pod \"payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.149152    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:38.154432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.149224    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:38.202411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.202377    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:12:38.207747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.207722    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.210761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.210724    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:12:38.309383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.309347    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:12:38.313413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.313372    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.313602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.313575    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzgv\" (UniqueName: \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.315524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.315500    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.321622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.321473    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:12:38.402433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.402400    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:38.415310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415264    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7vs\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415351    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415384    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415460    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415498    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415529    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415569    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.415705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415613    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.415705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415696    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzgv\" (UniqueName: \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.415965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.415948    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.416063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.416048    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.416518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.416491    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.439681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.439647    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzgv\" (UniqueName: \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\") pod \"payload-processing-e2e-api-a-3bda71-667649498b-bmj4r\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.517942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.517989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518074    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518109    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518146    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518182    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518345    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518460    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.518536    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7vs\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.519803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.519372    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.521461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.520864    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.521461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.521137    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.521461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.521408    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.525488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.525458    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.525990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.525962    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.527564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.527535    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.529111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.528866    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:38.544093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.544055    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7vs\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.547501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.547469    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pt8fh\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.639185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.639151    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:38.745781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.744068    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:12:38.752745 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:38.747256    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc763f895_9b4b_4e72_97cd_31f40459d067.slice/crio-30f6102648d2ff3c78c1d0b4cee385a98f6d7daf37281d4b156186eb97209d5c WatchSource:0}: Error finding container 30f6102648d2ff3c78c1d0b4cee385a98f6d7daf37281d4b156186eb97209d5c: Status 404 returned error can't find the container with id 30f6102648d2ff3c78c1d0b4cee385a98f6d7daf37281d4b156186eb97209d5c
Oct 30 07:12:38.900435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:38.900400    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:12:39.009479 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:39.009429    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266dea85_fd34_4eba_a344_f07f6fed8b8b.slice/crio-f82fd85a48b7fffce01bc74f74010f9e5adde19d3d589ef90e7de1ed4d2e120e WatchSource:0}: Error finding container f82fd85a48b7fffce01bc74f74010f9e5adde19d3d589ef90e7de1ed4d2e120e: Status 404 returned error can't find the container with id f82fd85a48b7fffce01bc74f74010f9e5adde19d3d589ef90e7de1ed4d2e120e
Oct 30 07:12:39.011471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.011118    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:12:39.012395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.012352    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:39.012471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.012453    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:39.012515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.012497    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:39.119192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.119155    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" event={"ID":"941ef547-a4e4-45d5-87c9-77b49d810231","Type":"ContainerStarted","Data":"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"}
Oct 30 07:12:39.119381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.119219    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" event={"ID":"941ef547-a4e4-45d5-87c9-77b49d810231","Type":"ContainerStarted","Data":"7eabebf8384724144a5dcd92ff7b57d5a6e6b69a45fe318cb6cef30dfbbf1877"}
Oct 30 07:12:39.119381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.119268    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:39.121192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.121166    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" event={"ID":"266dea85-fd34-4eba-a344-f07f6fed8b8b","Type":"ContainerStarted","Data":"f82fd85a48b7fffce01bc74f74010f9e5adde19d3d589ef90e7de1ed4d2e120e"}
Oct 30 07:12:39.124728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.123099    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" event={"ID":"c763f895-9b4b-4e72-97cd-31f40459d067","Type":"ContainerStarted","Data":"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"}
Oct 30 07:12:39.124728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.123147    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" event={"ID":"c763f895-9b4b-4e72-97cd-31f40459d067","Type":"ContainerStarted","Data":"30f6102648d2ff3c78c1d0b4cee385a98f6d7daf37281d4b156186eb97209d5c"}
Oct 30 07:12:39.124728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.123987    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:39.144200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.144163    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:39.144379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.144231    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:39.162633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.162254    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" podStartSLOduration=2.162235974 podStartE2EDuration="2.162235974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:39.161638648 +0000 UTC m=+919.594402536" watchObservedRunningTime="2026-10-30 07:12:39.162235974 +0000 UTC m=+919.594999867"
Oct 30 07:12:39.210331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.205619    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" podStartSLOduration=2.205599301 podStartE2EDuration="2.205599301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:39.205214198 +0000 UTC m=+919.637978105" watchObservedRunningTime="2026-10-30 07:12:39.205599301 +0000 UTC m=+919.638363194"
Oct 30 07:12:39.340037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:39.339849    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:40.210318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.202569    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:40.210318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.202625    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:40.316319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.310624    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" event={"ID":"266dea85-fd34-4eba-a344-f07f6fed8b8b","Type":"ContainerStarted","Data":"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"}
Oct 30 07:12:40.418695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.418612    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podStartSLOduration=2.418590367 podStartE2EDuration="2.418590367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:40.390794492 +0000 UTC m=+920.823558388" watchObservedRunningTime="2026-10-30 07:12:40.418590367 +0000 UTC m=+920.851354261"
Oct 30 07:12:40.639949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.639895    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:40.641679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.641646    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:40.641847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:40.641710    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:41.019327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.015811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:41.398331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.394758    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 07:12:41.398331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.394819    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 07:12:41.428318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.428062    6738 generic.go:356] "Generic (PLEG): container finished" podID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerID="de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d" exitCode=0
Oct 30 07:12:41.432320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.429359    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerDied","Data":"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"}
Oct 30 07:12:41.432320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.429823    6738 scope.go:122] "RemoveContainer" containerID="de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"
Oct 30 07:12:41.472331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.469444    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:41.472331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.469631    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:41.472331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.469691    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:41.741683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.645138    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:41.741683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:41.645220    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:42.118926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.118844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:42.438591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.438552    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerStarted","Data":"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"}
Oct 30 07:12:42.635018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.634983    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:42.635464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.635430    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:42.636160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.636139    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:12:42.640502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.640476    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:42.640601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.640526    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:42.766980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:42.766886    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:12:43.644318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:43.643423    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:43.644318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:43.643491    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:43.793827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:43.793440    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:43.793827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:43.793506    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:44.296933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:44.296891    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:44.641577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:44.641495    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:44.641787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:44.641601    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:44.767110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:44.767075    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:44.767692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:44.767136    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:45.640424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:45.640388    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:45.640609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:45.640442    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:45.740556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:45.740524    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:12:45.767018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:45.766985    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:45.767253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:45.767229    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:46.336559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:46.336517    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:46.640024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:46.639982    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:46.640210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:46.640057    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:46.766855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:46.766821    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:46.767041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:46.766875    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:47.640784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:47.640742    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pt8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 07:12:47.641205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:47.640814    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 07:12:47.767253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:47.767221    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:47.767489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:47.767283    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:48.308433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:48.308392    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:48.766723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:48.766688    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:12:48.767265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:48.766739    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 07:12:49.140166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:49.140130    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:49.140586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:49.140561    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:49.142239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:49.142213    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:12:49.771201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:49.771168    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:12:49.772368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:49.772345    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:12:50.274423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:50.274388    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:50.354491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:50.354461    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:12:51.432650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:51.432621    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:12:52.365229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:52.365187    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:52.487539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:52.487506    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:52.625536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:52.625497    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:54.676028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:54.675824    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:55.079695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:55.079595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:55.803302 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:55.803268    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:56.273163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.273125    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:12:56.279032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.279004    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.304834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.304779    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:12:56.308004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.307975    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7c7884-openshift-default-dockercfg-c8wsz\""
Oct 30 07:12:56.344615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.344574    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436717    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436769    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436806    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436860    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436939    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.436982    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgwz\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.437059    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.437085    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.446314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.437307    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.490441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.490391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:56.546306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.546194    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.546565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.546540    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.546729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.546713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.546849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.546835    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgwz\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.547005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.546991    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.547104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.547089    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.547389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.547372    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.547640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.547614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.547786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.547649    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.548860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.548125    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.548860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.548796    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.553486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.553460    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.554189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.554164    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.554462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.554408    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.577004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.576973    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.592167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.592097    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.592795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.592732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgwz\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.592954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.592796    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\") pod \"e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") " pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.602936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.602640    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:56.721473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.701930    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:56.838423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.837165    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"]
Oct 30 07:12:56.847441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.842277    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"]
Oct 30 07:12:56.847441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.842431    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:56.896460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.896404    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:12:56.896773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.896740    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:12:56.896955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.896938    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:12:56.897041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.896999    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:12:56.996923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.996889    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:56.997111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.997013    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:56.997196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:56.997149    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4pc\" (UniqueName: \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.098927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.098841    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.098927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.098914    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.099161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.098998    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4pc\" (UniqueName: \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.099748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.099720    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.101717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.101699    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.122565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.122529    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4pc\" (UniqueName: \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\") pod \"maas-api-revoke-sub-7ee2c980-cdb9g\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.146642 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:57.146598    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4264b8_a46c_4057_8fcb_65c23d1f1568.slice/crio-f3e37ae6369a96dd4d87fc3bfa4aad0f8d26196242dc2c628f3433459a42e782 WatchSource:0}: Error finding container f3e37ae6369a96dd4d87fc3bfa4aad0f8d26196242dc2c628f3433459a42e782: Status 404 returned error can't find the container with id f3e37ae6369a96dd4d87fc3bfa4aad0f8d26196242dc2c628f3433459a42e782
Oct 30 07:12:57.148782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.148748    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:12:57.149751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.149717    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:57.149822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.149792    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:57.150353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.150317    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:12:57.233432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.233405    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:12:57.550963 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:12:57.550924    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14408016_3c28_4a4e_bc94_e3122d4ad5d1.slice/crio-6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797 WatchSource:0}: Error finding container 6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797: Status 404 returned error can't find the container with id 6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797
Oct 30 07:12:57.553020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.553003    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"]
Oct 30 07:12:57.782565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.782517    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerStarted","Data":"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"}
Oct 30 07:12:57.782769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.782588    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerStarted","Data":"f3e37ae6369a96dd4d87fc3bfa4aad0f8d26196242dc2c628f3433459a42e782"}
Oct 30 07:12:57.783994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.783970    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g" event={"ID":"14408016-3c28-4a4e-bc94-e3122d4ad5d1","Type":"ContainerStarted","Data":"6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797"}
Oct 30 07:12:57.831766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:57.831641    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podStartSLOduration=1.831623226 podStartE2EDuration="1.831623226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:57.831398986 +0000 UTC m=+938.264162878" watchObservedRunningTime="2026-10-30 07:12:57.831623226 +0000 UTC m=+938.264387119"
Oct 30 07:12:58.603342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.603310    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:58.610186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.610160    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:58.795233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.795195    6738 generic.go:356] "Generic (PLEG): container finished" podID="14408016-3c28-4a4e-bc94-e3122d4ad5d1" containerID="4969041ac7d2640d3bf671f3043b473cb0ce805fb50167a69bd0d7512798d5c1" exitCode=0
Oct 30 07:12:58.795417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.795277    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g" event={"ID":"14408016-3c28-4a4e-bc94-e3122d4ad5d1","Type":"ContainerDied","Data":"4969041ac7d2640d3bf671f3043b473cb0ce805fb50167a69bd0d7512798d5c1"}
Oct 30 07:12:58.795799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.795775    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:58.796703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.796680    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:12:58.847979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:58.847884    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:12:59.966883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:12:59.966858    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:13:00.056960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.056867    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\" (UniqueName: \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\") pod \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") "
Oct 30 07:13:00.057127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.056966    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\") pod \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") "
Oct 30 07:13:00.057127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.057004    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\") pod \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\" (UID: \"14408016-3c28-4a4e-bc94-e3122d4ad5d1\") "
Oct 30 07:13:00.057627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.057502    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca" pod "14408016-3c28-4a4e-bc94-e3122d4ad5d1" (UID: "14408016-3c28-4a4e-bc94-e3122d4ad5d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:13:00.057882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.057854    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:00.059270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.059246    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc" pod "14408016-3c28-4a4e-bc94-e3122d4ad5d1" (UID: "14408016-3c28-4a4e-bc94-e3122d4ad5d1"). InnerVolumeSpecName "kube-api-access-5q4pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:13:00.059513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.059484    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca" pod "14408016-3c28-4a4e-bc94-e3122d4ad5d1" (UID: "14408016-3c28-4a4e-bc94-e3122d4ad5d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:13:00.159415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.159379    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4pc\" (UniqueName: \"kubernetes.io/projected/14408016-3c28-4a4e-bc94-e3122d4ad5d1-kube-api-access-5q4pc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:00.159415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.159408    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14408016-3c28-4a4e-bc94-e3122d4ad5d1-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:00.294572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.294527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:00.420575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.420540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:00.812149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.812057    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"
Oct 30 07:13:00.812149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.812073    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g" event={"ID":"14408016-3c28-4a4e-bc94-e3122d4ad5d1","Type":"ContainerDied","Data":"6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797"}
Oct 30 07:13:00.812149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.812123    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6f2d806171a352d64d47d5510578bfe0de5c95b43949945b65a7190700a797"
Oct 30 07:13:00.983797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:00.983765    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:02.827246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:02.827181    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:02.948929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:02.948888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:03.010804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.010759    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:13:03.011909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.011882    6738 cpu_manager.go:414] "containerMap: removing container" podUID="14408016-3c28-4a4e-bc94-e3122d4ad5d1" containerName="revoke-keys"
Oct 30 07:13:03.012004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.011912    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14408016-3c28-4a4e-bc94-e3122d4ad5d1" containerName="revoke-keys"
Oct 30 07:13:03.012227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.012087    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="14408016-3c28-4a4e-bc94-e3122d4ad5d1" containerName="revoke-keys"
Oct 30 07:13:03.015963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.015938    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.034893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.034860    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:13:03.040154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.040123    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7c7884\""
Oct 30 07:13:03.046840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.046789    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7c7884-dockercfg-rn87z\""
Oct 30 07:13:03.232704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.176035    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.232704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.176182    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9c56\" (UniqueName: \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.325334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.316136    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9c56\" (UniqueName: \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.325334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.316465    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.325334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.317568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.430361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.400896    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9c56\" (UniqueName: \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\") pod \"payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.434193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.432494    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:13:03.439538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.436687    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:13:03.439538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.436813    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.534026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.533936    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm4ff\" (UniqueName: \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.534187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.534164    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.621121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.621082    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:03.635792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.635756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm4ff\" (UniqueName: \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.636018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.635996    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.636269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.636247    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:03.636859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.636838    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:03.778317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:03.777678    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm4ff\" (UniqueName: \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\") pod \"payload-processing-e2e-api-b-7c7884-56db767db-h7gsr\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:04.058924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.058892    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:04.099591 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:04.099542    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d14e0a8_699e_48a2_879b_b3a9ab9e771a.slice/crio-4963297fb25ef9e0d5e13d9e50dc24da986f0466a0e0445ea530f02e5ff2faee WatchSource:0}: Error finding container 4963297fb25ef9e0d5e13d9e50dc24da986f0466a0e0445ea530f02e5ff2faee: Status 404 returned error can't find the container with id 4963297fb25ef9e0d5e13d9e50dc24da986f0466a0e0445ea530f02e5ff2faee
Oct 30 07:13:04.101004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.100979    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:13:04.399247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.399219    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:13:04.401756 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:04.401731    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd522eb1_eda2_4129_8929_b599abf2712f.slice/crio-a96d59e620366d25cedf68f8f30521a6e2ad2bbb2e9eb1a1a533bda7853f0f73 WatchSource:0}: Error finding container a96d59e620366d25cedf68f8f30521a6e2ad2bbb2e9eb1a1a533bda7853f0f73: Status 404 returned error can't find the container with id a96d59e620366d25cedf68f8f30521a6e2ad2bbb2e9eb1a1a533bda7853f0f73
Oct 30 07:13:04.897067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.897028    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" event={"ID":"dd522eb1-eda2-4129-8929-b599abf2712f","Type":"ContainerStarted","Data":"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"}
Oct 30 07:13:04.897284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.897096    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" event={"ID":"dd522eb1-eda2-4129-8929-b599abf2712f","Type":"ContainerStarted","Data":"a96d59e620366d25cedf68f8f30521a6e2ad2bbb2e9eb1a1a533bda7853f0f73"}
Oct 30 07:13:04.897284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.897123    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:04.898812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.898786    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" event={"ID":"8d14e0a8-699e-48a2-879b-b3a9ab9e771a","Type":"ContainerStarted","Data":"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"}
Oct 30 07:13:04.898925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.898829    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" event={"ID":"8d14e0a8-699e-48a2-879b-b3a9ab9e771a","Type":"ContainerStarted","Data":"4963297fb25ef9e0d5e13d9e50dc24da986f0466a0e0445ea530f02e5ff2faee"}
Oct 30 07:13:04.898925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.898914    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:04.939283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.939231    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" podStartSLOduration=1.939216414 podStartE2EDuration="1.939216414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:04.937618516 +0000 UTC m=+945.370382408" watchObservedRunningTime="2026-10-30 07:13:04.939216414 +0000 UTC m=+945.371980348"
Oct 30 07:13:04.994544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:04.994237    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" podStartSLOduration=2.994212508 podStartE2EDuration="2.994212508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:04.983643315 +0000 UTC m=+945.416407236" watchObservedRunningTime="2026-10-30 07:13:04.994212508 +0000 UTC m=+945.426976404"
Oct 30 07:13:05.852356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:05.852324    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:07.069349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.067605    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"]
Oct 30 07:13:07.095316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.095230    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.101021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.099985    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"]
Oct 30 07:13:07.112386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.112354    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:13:07.112570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.112397    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:13:07.112649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.112623    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:13:07.116396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.116368    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.116529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.116409    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.116529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.116470    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljt4\" (UniqueName: \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.125046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.125024    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:13:07.218020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.217985    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.218020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.218020    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.218254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.218039    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljt4\" (UniqueName: \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.218702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.218675    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.220460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.220437    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.241599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.241572    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljt4\" (UniqueName: \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\") pod \"maas-api-revoke-sub-8a49e384-bltmg\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.416980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.416939    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:07.539902 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:13:07.539842    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:13:07.756253 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:07.756221    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fb4366_c3f5_4f6c_a94c_bfbc80d505c2.slice/crio-3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13 WatchSource:0}: Error finding container 3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13: Status 404 returned error can't find the container with id 3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13
Oct 30 07:13:07.757801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.757779    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"]
Oct 30 07:13:07.856836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:07.856785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:08.031371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:08.030813    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg" event={"ID":"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2","Type":"ContainerStarted","Data":"3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13"}
Oct 30 07:13:09.040263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:09.040226    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg" event={"ID":"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2","Type":"ContainerStarted","Data":"8858d0e1809ac2a262755601b24ca5a1583648c5f168daf1d7dff4d63d373952"}
Oct 30 07:13:09.095070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:09.094579    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg" podStartSLOduration=2.094562748 podStartE2EDuration="2.094562748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:09.084509905 +0000 UTC m=+949.517273800" watchObservedRunningTime="2026-10-30 07:13:09.094562748 +0000 UTC m=+949.527326641"
Oct 30 07:13:09.837228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:09.837186    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:10.050362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:10.050323    6738 generic.go:356] "Generic (PLEG): container finished" podID="c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" containerID="8858d0e1809ac2a262755601b24ca5a1583648c5f168daf1d7dff4d63d373952" exitCode=0
Oct 30 07:13:10.050778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:10.050458    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg" event={"ID":"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2","Type":"ContainerDied","Data":"8858d0e1809ac2a262755601b24ca5a1583648c5f168daf1d7dff4d63d373952"}
Oct 30 07:13:11.261528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.261500    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:11.391768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.391721    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\" (UniqueName: \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\") pod \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") "
Oct 30 07:13:11.391984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.391893    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\") pod \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") "
Oct 30 07:13:11.391984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.391943    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\") pod \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\" (UID: \"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2\") "
Oct 30 07:13:11.392327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.392278    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca" pod "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" (UID: "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:13:11.392657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.392634    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:11.393991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.393961    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4" pod "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" (UID: "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2"). InnerVolumeSpecName "kube-api-access-tljt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:13:11.394489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.394461    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca" pod "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" (UID: "c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:13:11.493786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.493754    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljt4\" (UniqueName: \"kubernetes.io/projected/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-kube-api-access-tljt4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:11.493786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.493783    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:11.734777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:11.734734    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:12.082441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:12.082363    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"
Oct 30 07:13:12.082441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:12.082405    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg" event={"ID":"c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2","Type":"ContainerDied","Data":"3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13"}
Oct 30 07:13:12.082655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:12.082463    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3494f8d8681ad2a9e2a91996ec6130986690b2a7215e4be5fa107b96415e0d13"
Oct 30 07:13:13.092279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.092189    6738 generic.go:356] "Generic (PLEG): container finished" podID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerID="6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501" exitCode=0
Oct 30 07:13:13.092279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.092265    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerDied","Data":"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"}
Oct 30 07:13:13.092873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.092740    6738 scope.go:122] "RemoveContainer" containerID="6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"
Oct 30 07:13:13.094506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.094476    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:13.094597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.094548    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:13.094597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.094585    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:13.794326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:13.794264    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:14.103228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:14.103136    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerStarted","Data":"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"}
Oct 30 07:13:14.404424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:14.404393    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:13:14.405731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:14.405700    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:14.405869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:14.405751    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:15.404076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:15.404039    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:15.404558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:15.404095    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:15.834179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:15.834079    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:15.988596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:15.988565    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:13:16.008261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:16.008232    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:13:16.126228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:16.126172    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:16.404977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:16.404890    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:16.404977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:16.404961    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:16.852237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:16.852137    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:17.399862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:17.399831    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:17.404752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:17.404721    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:17.404984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:17.404961    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:17.548932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:17.548894    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:18.190213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:18.190178    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:18.403804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:18.403756    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:18.404009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:18.403820    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:18.748066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:18.748028    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:13:19.404542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:19.404500    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:13:19.404735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:19.404573    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 07:13:20.095790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.095750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:20.137203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.137164    6738 scope.go:122] "RemoveContainer" containerID="1efd0495a6642a4a1c958c1a3947afbeea4e4d59be30286562d9dfc630b0f1ea"
Oct 30 07:13:20.149518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.149497    6738 scope.go:122] "RemoveContainer" containerID="dcf5d44236b5eb98486cbd469b07d991c5d297e011c929619163a21a1a21dde3"
Oct 30 07:13:20.217379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.217202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:20.371909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.371874    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:20.409083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.409056    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:13:20.410501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:20.410478    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:13:21.586379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:21.586342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:21.706599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:21.706560    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:22.407797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:22.407747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:23.492693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:23.492656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:23.679495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:23.633117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:24.526787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.526745    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:13:24.528374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.528332    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" containerName="revoke-keys"
Oct 30 07:13:24.528681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.528642    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" containerName="revoke-keys"
Oct 30 07:13:24.529208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.529088    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" containerName="revoke-keys"
Oct 30 07:13:24.554383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.552699    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.588354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.588270    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f357db-dockercfg-qwc79\""
Oct 30 07:13:24.588792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.588771    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f357db\""
Oct 30 07:13:24.590032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.590011    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:13:24.600808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.600774    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:13:24.637578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.637545    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:13:24.637963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.637941    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.671984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.671939    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6dl\" (UniqueName: \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.672179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.672058    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.672261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.672182    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.672261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.672239    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5jx\" (UniqueName: \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.773425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.773370    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6dl\" (UniqueName: \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.773637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.773505    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.773720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.773646    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.773720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.773707    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5jx\" (UniqueName: \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.775094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.775049    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.775449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.775424    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.801568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.801360    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5jx\" (UniqueName: \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\") pod \"payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.801568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.801363    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6dl\" (UniqueName: \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\") pod \"payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:24.894319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.893561    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:24.979717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:24.979619    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:25.415316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:25.408681    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:25.699368 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:25.688927    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ebf63a_2d38_425c_bf18_15840044abdf.slice/crio-49db20e22351f19bf3c330f832c62b8a4fe1dc241c534868ece87973cd0bfaca WatchSource:0}: Error finding container 49db20e22351f19bf3c330f832c62b8a4fe1dc241c534868ece87973cd0bfaca: Status 404 returned error can't find the container with id 49db20e22351f19bf3c330f832c62b8a4fe1dc241c534868ece87973cd0bfaca
Oct 30 07:13:25.717698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:25.717665    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:13:25.966619 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:25.966577    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d145b9_bc63_4c7d_9acf_04cefa82cb07.slice/crio-a06c233377e336ccc0790835ea39e0e2991493eaf6a403eda129cbc9ed19b561 WatchSource:0}: Error finding container a06c233377e336ccc0790835ea39e0e2991493eaf6a403eda129cbc9ed19b561: Status 404 returned error can't find the container with id a06c233377e336ccc0790835ea39e0e2991493eaf6a403eda129cbc9ed19b561
Oct 30 07:13:25.967312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:25.967274    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:13:26.606110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.606073    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" event={"ID":"25ebf63a-2d38-425c-bf18-15840044abdf","Type":"ContainerStarted","Data":"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"}
Oct 30 07:13:26.606336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.606130    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" event={"ID":"25ebf63a-2d38-425c-bf18-15840044abdf","Type":"ContainerStarted","Data":"49db20e22351f19bf3c330f832c62b8a4fe1dc241c534868ece87973cd0bfaca"}
Oct 30 07:13:26.606336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.606175    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:26.608039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.608016    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" event={"ID":"52d145b9-bc63-4c7d-9acf-04cefa82cb07","Type":"ContainerStarted","Data":"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"}
Oct 30 07:13:26.608125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.608057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" event={"ID":"52d145b9-bc63-4c7d-9acf-04cefa82cb07","Type":"ContainerStarted","Data":"a06c233377e336ccc0790835ea39e0e2991493eaf6a403eda129cbc9ed19b561"}
Oct 30 07:13:26.608125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.608108    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:26.649679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.649623    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" podStartSLOduration=3.649607389 podStartE2EDuration="3.649607389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:26.646581347 +0000 UTC m=+967.079345240" watchObservedRunningTime="2026-10-30 07:13:26.649607389 +0000 UTC m=+967.082371279"
Oct 30 07:13:26.687878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:26.687796    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" podStartSLOduration=3.68778629 podStartE2EDuration="3.68778629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:26.687081035 +0000 UTC m=+967.119844929" watchObservedRunningTime="2026-10-30 07:13:26.68778629 +0000 UTC m=+967.120550184"
Oct 30 07:13:27.427011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:27.426977    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:28.936594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:28.936539    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:29.055070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:29.055030    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"]
Oct 30 07:13:29.059827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:29.059797    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb7fc9e-vbgjf"]
Oct 30 07:13:29.276595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:29.276509    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:29.691963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:29.691933    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162da9e3-dc50-4df5-b4b6-afdd779d5708" path="/var/lib/kubelet/pods/162da9e3-dc50-4df5-b4b6-afdd779d5708/volumes"
Oct 30 07:13:31.240999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:31.240955    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:33.060842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:33.060798    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:35.143735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:35.143690    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:37.133847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:37.133807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:37.519701 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:13:37.519619    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:13:37.692315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:37.678601    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:13:37.692315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:37.683894    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:13:38.248623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:38.248588    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:39.137307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:39.137252    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:41.046806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:41.046771    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:43.056647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.056605    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:43.195314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.185389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:43.308365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.307366    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"]
Oct 30 07:13:43.329832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.329793    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.342266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.342232    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"]
Oct 30 07:13:43.361694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.347030    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:13:43.361694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.347182    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:13:43.361849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.361695    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:13:43.373781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.362448    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:13:43.465777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.465743    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.465949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.465825    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.466138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.466111    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gphx\" (UniqueName: \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.568975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.568010    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.569514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.569488    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gphx\" (UniqueName: \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.570190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.570168    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.570926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.568899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.596771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.596736    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.599688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.599660    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gphx\" (UniqueName: \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\") pod \"maas-api-revoke-sub-83fbb16e-7r4qh\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:43.712190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:43.712161    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:44.079317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.071347    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"]
Oct 30 07:13:44.095321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.094307    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37907a09-ctgrh"]
Oct 30 07:13:44.097215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.097193    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"]
Oct 30 07:13:44.105269 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:44.105238    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3755e299_809e_4374_b35e_7a6bc5a7c032.slice/crio-f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17 WatchSource:0}: Error finding container f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17: Status 404 returned error can't find the container with id f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17
Oct 30 07:13:44.612345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.560352    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:13:44.622556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.621593    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.803320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.782761    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.803668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.803620    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.803980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.803955    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.804175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.804153    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kc5w\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.828550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.824228    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:13:44.828826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.828775    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.829002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.828986    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.829128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.829112    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.829287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.829271    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.829430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.829415    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932415    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932465    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932488    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932520    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932677    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932698    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932773    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.932905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.932808    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kc5w\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.933956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.933922    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.934465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.934437    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.934770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.934748    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.935051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.935027    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.936265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.936209    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.938195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.938175    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.939283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.939261    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.995315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.995269    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:44.995502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:44.995381    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kc5w\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:45.107986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.107950    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh" event={"ID":"3755e299-809e-4374-b35e-7a6bc5a7c032","Type":"ContainerStarted","Data":"3fb3545e8d8eb3c6993881fffbeb1c4482b8c2f5eb6079d3e213e3dcb883d1ed"}
Oct 30 07:13:45.108393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.108006    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh" event={"ID":"3755e299-809e-4374-b35e-7a6bc5a7c032","Type":"ContainerStarted","Data":"f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17"}
Oct 30 07:13:45.135176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.135138    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:45.161036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.160979    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh" podStartSLOduration=2.160964328 podStartE2EDuration="2.160964328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:45.157437621 +0000 UTC m=+985.590201514" watchObservedRunningTime="2026-10-30 07:13:45.160964328 +0000 UTC m=+985.593728220"
Oct 30 07:13:45.366982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.366948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:45.553493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.553399    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:13:45.558395 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:45.558366    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14df479_780b_4d3a_8844_e2f1449c75ed.slice/crio-b99614409e0a9589a94933df6c1a32911ad5682c7062a9e58adb7d9be9eb7f0e WatchSource:0}: Error finding container b99614409e0a9589a94933df6c1a32911ad5682c7062a9e58adb7d9be9eb7f0e: Status 404 returned error can't find the container with id b99614409e0a9589a94933df6c1a32911ad5682c7062a9e58adb7d9be9eb7f0e
Oct 30 07:13:45.561160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.561122    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:45.561269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.561197    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:45.561269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.561227    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:45.708429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:45.708353    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d989e68-b813-43f7-934c-01fdb8a8bfed" path="/var/lib/kubelet/pods/8d989e68-b813-43f7-934c-01fdb8a8bfed/volumes"
Oct 30 07:13:46.119006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.118960    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" event={"ID":"e14df479-780b-4d3a-8844-e2f1449c75ed","Type":"ContainerStarted","Data":"e80bd7a42c3d1cc23ca09872afc6e9dd94ea90db8aed1245a6282131d0d5c21d"}
Oct 30 07:13:46.119538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.119039    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" event={"ID":"e14df479-780b-4d3a-8844-e2f1449c75ed","Type":"ContainerStarted","Data":"b99614409e0a9589a94933df6c1a32911ad5682c7062a9e58adb7d9be9eb7f0e"}
Oct 30 07:13:46.121633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.121605    6738 generic.go:356] "Generic (PLEG): container finished" podID="3755e299-809e-4374-b35e-7a6bc5a7c032" containerID="3fb3545e8d8eb3c6993881fffbeb1c4482b8c2f5eb6079d3e213e3dcb883d1ed" exitCode=0
Oct 30 07:13:46.121779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.121681    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh" event={"ID":"3755e299-809e-4374-b35e-7a6bc5a7c032","Type":"ContainerDied","Data":"3fb3545e8d8eb3c6993881fffbeb1c4482b8c2f5eb6079d3e213e3dcb883d1ed"}
Oct 30 07:13:46.136375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.136347    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:46.137750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.137719    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:46.137888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.137772    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:46.191315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.191078    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podStartSLOduration=2.191057336 podStartE2EDuration="2.191057336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:46.162507257 +0000 UTC m=+986.595271149" watchObservedRunningTime="2026-10-30 07:13:46.191057336 +0000 UTC m=+986.623821229"
Oct 30 07:13:46.194455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:46.194373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:47.135440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.135400    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:47.135909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.135456    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:47.440873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.440835    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:47.469329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.459203    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:47.655410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.646921    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\" (UniqueName: \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\") pod \"3755e299-809e-4374-b35e-7a6bc5a7c032\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") "
Oct 30 07:13:47.655410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.647003    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\") pod \"3755e299-809e-4374-b35e-7a6bc5a7c032\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") "
Oct 30 07:13:47.655410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.647061    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\") pod \"3755e299-809e-4374-b35e-7a6bc5a7c032\" (UID: \"3755e299-809e-4374-b35e-7a6bc5a7c032\") "
Oct 30 07:13:47.677534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.665852    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca" pod "3755e299-809e-4374-b35e-7a6bc5a7c032" (UID: "3755e299-809e-4374-b35e-7a6bc5a7c032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:13:47.734552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.733003    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca" pod "3755e299-809e-4374-b35e-7a6bc5a7c032" (UID: "3755e299-809e-4374-b35e-7a6bc5a7c032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:13:47.734552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.734431    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx" pod "3755e299-809e-4374-b35e-7a6bc5a7c032" (UID: "3755e299-809e-4374-b35e-7a6bc5a7c032"). InnerVolumeSpecName "kube-api-access-9gphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:13:47.748629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.748599    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gphx\" (UniqueName: \"kubernetes.io/projected/3755e299-809e-4374-b35e-7a6bc5a7c032-kube-api-access-9gphx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:47.748629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.748627    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:47.748834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:47.748642    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3755e299-809e-4374-b35e-7a6bc5a7c032-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:48.137076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:48.135974    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:48.137076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:48.136053    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:48.145867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:48.145838    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"
Oct 30 07:13:48.146042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:48.145950    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh" event={"ID":"3755e299-809e-4374-b35e-7a6bc5a7c032","Type":"ContainerDied","Data":"f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17"}
Oct 30 07:13:48.146042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:48.145996    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a26d1a2ed24108b45591bedcff797cc092aa2cf37ba1eaa3478fe67a363f17"
Oct 30 07:13:49.105857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.105763    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"]
Oct 30 07:13:49.106629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.106605    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3755e299-809e-4374-b35e-7a6bc5a7c032" containerName="revoke-keys"
Oct 30 07:13:49.106629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.106628    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3755e299-809e-4374-b35e-7a6bc5a7c032" containerName="revoke-keys"
Oct 30 07:13:49.106855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.106841    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3755e299-809e-4374-b35e-7a6bc5a7c032" containerName="revoke-keys"
Oct 30 07:13:49.110024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.109998    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.119845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.119813    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"]
Oct 30 07:13:49.125672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.125642    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:13:49.135647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.135613    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:49.135816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.135669    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:49.137642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.137615    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:13:49.137642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.137624    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:13:49.138015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.137626    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:13:49.200344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.200284    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.200888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.200861    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlw7\" (UniqueName: \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.201057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.201040    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.303373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.303333    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlw7\" (UniqueName: \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.303572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.303380    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.303572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.303510    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.304326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.304284    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.306279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.306247    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.326965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.326934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlw7\" (UniqueName: \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\") pod \"maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.428448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.428412    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:49.656509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.656478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:49.883336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:49.861864    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"]
Oct 30 07:13:50.136343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:50.136288    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:50.136541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:50.136381    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:50.165080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:50.165046    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4" event={"ID":"e423ee77-f9b3-417a-9a80-8cb5282b0667","Type":"ContainerStarted","Data":"2a4a17b27294091f506d06b575276e3618d8e8c59e319d704fd41071ca92d173"}
Oct 30 07:13:51.136157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:51.136120    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:51.136375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:51.136195    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:51.203368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:51.203329    6738 generic.go:356] "Generic (PLEG): container finished" podID="e423ee77-f9b3-417a-9a80-8cb5282b0667" containerID="ce4db065c70bd81dc4deed9ad17bab8e26ac0d798b13cb329fb043924ad24844" exitCode=0
Oct 30 07:13:51.203876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:51.203411    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4" event={"ID":"e423ee77-f9b3-417a-9a80-8cb5282b0667","Type":"ContainerDied","Data":"ce4db065c70bd81dc4deed9ad17bab8e26ac0d798b13cb329fb043924ad24844"}
Oct 30 07:13:52.104711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.103156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:52.136487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.136445    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:52.162031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.161988    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:52.539739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.539367    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:52.657527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.657439    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\") pod \"e423ee77-f9b3-417a-9a80-8cb5282b0667\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") "
Oct 30 07:13:52.657823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.657801    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\" (UniqueName: \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\") pod \"e423ee77-f9b3-417a-9a80-8cb5282b0667\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") "
Oct 30 07:13:52.657996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.657978    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\") pod \"e423ee77-f9b3-417a-9a80-8cb5282b0667\" (UID: \"e423ee77-f9b3-417a-9a80-8cb5282b0667\") "
Oct 30 07:13:52.658693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.658668    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca" pod "e423ee77-f9b3-417a-9a80-8cb5282b0667" (UID: "e423ee77-f9b3-417a-9a80-8cb5282b0667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:13:52.659035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.659013    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:52.662730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.662666    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca" pod "e423ee77-f9b3-417a-9a80-8cb5282b0667" (UID: "e423ee77-f9b3-417a-9a80-8cb5282b0667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:13:52.663025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.662998    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7" pod "e423ee77-f9b3-417a-9a80-8cb5282b0667" (UID: "e423ee77-f9b3-417a-9a80-8cb5282b0667"). InnerVolumeSpecName "kube-api-access-6dlw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:13:52.760623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.760581    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e423ee77-f9b3-417a-9a80-8cb5282b0667-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:52.760623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:52.760624    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlw7\" (UniqueName: \"kubernetes.io/projected/e423ee77-f9b3-417a-9a80-8cb5282b0667-kube-api-access-6dlw7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:13:53.137047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:53.137010    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 07:13:53.137345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:53.137313    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 07:13:53.353784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:53.353466    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"
Oct 30 07:13:53.358369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:53.353466    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4" event={"ID":"e423ee77-f9b3-417a-9a80-8cb5282b0667","Type":"ContainerDied","Data":"2a4a17b27294091f506d06b575276e3618d8e8c59e319d704fd41071ca92d173"}
Oct 30 07:13:53.358659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:53.358635    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4a17b27294091f506d06b575276e3618d8e8c59e319d704fd41071ca92d173"
Oct 30 07:13:54.071397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.071346    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"]
Oct 30 07:13:54.082326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.079791    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9900ca1-rvx6s"]
Oct 30 07:13:54.289319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.284385    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:13:54.519320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.515633    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:54.519320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.517446    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:54.540834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.540792    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:13:54.770127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:54.770039    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:55.045012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:55.044896    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"]
Oct 30 07:13:55.055585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:55.055546    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc002b8a-lbjnh"]
Oct 30 07:13:55.694154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:55.694113    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1814cc7d-3186-4c87-bc24-1646213dc06b" path="/var/lib/kubelet/pods/1814cc7d-3186-4c87-bc24-1646213dc06b/volumes"
Oct 30 07:13:55.694895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:55.694874    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9" path="/var/lib/kubelet/pods/39d70618-8d1a-4ae6-a5e0-ee893ae9e3b9/volumes"
Oct 30 07:13:56.112208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.112112    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:56.443926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.443893    6738 generic.go:356] "Generic (PLEG): container finished" podID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerID="ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8" exitCode=0
Oct 30 07:13:56.444236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.443967    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerDied","Data":"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"}
Oct 30 07:13:56.444466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.444449    6738 scope.go:122] "RemoveContainer" containerID="ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"
Oct 30 07:13:56.445963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.445933    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:56.446025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.446003    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:56.446064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.446035    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:56.562899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.562860    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:13:56.564679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.564024    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e423ee77-f9b3-417a-9a80-8cb5282b0667" containerName="revoke-keys"
Oct 30 07:13:56.564679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.564051    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e423ee77-f9b3-417a-9a80-8cb5282b0667" containerName="revoke-keys"
Oct 30 07:13:56.564679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.564205    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e423ee77-f9b3-417a-9a80-8cb5282b0667" containerName="revoke-keys"
Oct 30 07:13:56.567509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.567481    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.593043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.589930    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c0e3de-openshift-default-dockercfg-ww47w\""
Oct 30 07:13:56.597370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.597121    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:13:56.642990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.642951    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:56.663360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663326    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663379    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663498    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663571    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kvv\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663648    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663721    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663807    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663839    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.663959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.663948    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.763180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.763090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:56.765700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.765673    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.765802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.765712    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.765854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.765829    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.765897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.765889    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.765980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.765964    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.766045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.766008    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.766383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.766355    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.766794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.766771    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.766898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.766856    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kvv\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.767050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.767026    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.767197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.767124    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.767197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.767176    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.767583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.767561    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.767704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.767678    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.768341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.768277    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.768687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.768670    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.789701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.789666    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.790173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.790148    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kvv\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\") pod \"e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") " pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:56.900313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.895234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:56.903916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:56.903620    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:57.100304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.099515    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"]
Oct 30 07:13:57.112390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.112348    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9f912e-9fzkr"]
Oct 30 07:13:57.456615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.456547    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerStarted","Data":"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"}
Oct 30 07:13:57.506661 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:57.506632    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7495b90f_0807_44c4_a023_7dfe4107408b.slice/crio-c0e74d16fa995f8c6282150b28b1efc9f383f7c0e102ecaf0986b87c06d2b002 WatchSource:0}: Error finding container c0e74d16fa995f8c6282150b28b1efc9f383f7c0e102ecaf0986b87c06d2b002: Status 404 returned error can't find the container with id c0e74d16fa995f8c6282150b28b1efc9f383f7c0e102ecaf0986b87c06d2b002
Oct 30 07:13:57.508232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.508206    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:13:57.510129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.510093    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:57.510242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.510174    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:57.510242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.510223    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:13:57.603523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.603487    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:13:57.604953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.604923    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:13:57.605076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.604980    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:13:57.711476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.711372    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a40e5c-7647-4525-9cf9-276a20f00509" path="/var/lib/kubelet/pods/01a40e5c-7647-4525-9cf9-276a20f00509/volumes"
Oct 30 07:13:57.903604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.899891    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"]
Oct 30 07:13:57.906323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.906232    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:57.921514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.921485    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"]
Oct 30 07:13:57.922536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.922508    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:13:57.922649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.922629    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:13:57.922754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.922730    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:13:57.922824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:57.922781    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:13:58.008951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.008846    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.009149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.009100    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.009234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.009213    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9vz\" (UniqueName: \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.111053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.111018    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.111249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.111164    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.111249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.111232    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9vz\" (UniqueName: \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.111896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.111868    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.113555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.113538    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.135136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.135099    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9vz\" (UniqueName: \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.221440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.221401    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:13:58.498321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.496635    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerStarted","Data":"a3bee4efcf6e7664063ba3be38ecb7a64d0a2fd231f2e93af252daaf868ed781"}
Oct 30 07:13:58.498321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.496696    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerStarted","Data":"c0e74d16fa995f8c6282150b28b1efc9f383f7c0e102ecaf0986b87c06d2b002"}
Oct 30 07:13:58.546997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.546940    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podStartSLOduration=2.546924604 podStartE2EDuration="2.546924604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:58.543465407 +0000 UTC m=+998.976229300" watchObservedRunningTime="2026-10-30 07:13:58.546924604 +0000 UTC m=+998.979688572"
Oct 30 07:13:58.560940 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:13:58.560910    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72c2c14_562e_4616_8c28_c7da12b45d22.slice/crio-607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98 WatchSource:0}: Error finding container 607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98: Status 404 returned error can't find the container with id 607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98
Oct 30 07:13:58.562974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.562946    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"]
Oct 30 07:13:58.604132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.604098    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:13:58.604316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.604152    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:13:58.807373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.796404    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:13:58.822456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.820865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:58.927325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.926124    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:58.978348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:58.970757    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:59.509742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.509645    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc" event={"ID":"d72c2c14-562e-4616-8c28-c7da12b45d22","Type":"ContainerStarted","Data":"b43b463f2c3adc3b31d3dcfc80a3a9ef8975bd6d235b8ce73f8d38c8d9fc5c8d"}
Oct 30 07:13:59.509742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.509732    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc" event={"ID":"d72c2c14-562e-4616-8c28-c7da12b45d22","Type":"ContainerStarted","Data":"607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98"}
Oct 30 07:13:59.510286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.510262    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:59.511181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.511158    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:13:59.561555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.561499    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc" podStartSLOduration=2.561485063 podStartE2EDuration="2.561485063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:59.556924666 +0000 UTC m=+999.989688559" watchObservedRunningTime="2026-10-30 07:13:59.561485063 +0000 UTC m=+999.994248957"
Oct 30 07:13:59.604345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.604308    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:13:59.604546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.604366    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:13:59.615870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.615835    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:13:59.725639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.725597    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:13:59.957746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:13:59.953015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:00.573318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:00.565817    6738 generic.go:356] "Generic (PLEG): container finished" podID="d72c2c14-562e-4616-8c28-c7da12b45d22" containerID="b43b463f2c3adc3b31d3dcfc80a3a9ef8975bd6d235b8ce73f8d38c8d9fc5c8d" exitCode=0
Oct 30 07:14:00.573318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:00.567464    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc" event={"ID":"d72c2c14-562e-4616-8c28-c7da12b45d22","Type":"ContainerDied","Data":"b43b463f2c3adc3b31d3dcfc80a3a9ef8975bd6d235b8ce73f8d38c8d9fc5c8d"}
Oct 30 07:14:00.604189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:00.604152    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:14:00.604419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:00.604213    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:14:00.961979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:00.961940    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:01.127608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.127568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:01.394067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.394028    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:01.603817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.603782    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:14:01.604014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.603838    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:14:01.640583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.640463    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:01.822103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:01.821963    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:02.051227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.051038    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:02.302447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.302419    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:14:02.322743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.322707    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\") pod \"d72c2c14-562e-4616-8c28-c7da12b45d22\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") "
Oct 30 07:14:02.322923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.322832    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\" (UniqueName: \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\") pod \"d72c2c14-562e-4616-8c28-c7da12b45d22\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") "
Oct 30 07:14:02.322923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.322889    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\") pod \"d72c2c14-562e-4616-8c28-c7da12b45d22\" (UID: \"d72c2c14-562e-4616-8c28-c7da12b45d22\") "
Oct 30 07:14:02.323090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.323068    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca" pod "d72c2c14-562e-4616-8c28-c7da12b45d22" (UID: "d72c2c14-562e-4616-8c28-c7da12b45d22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:02.323604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.323582    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:02.324991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.324966    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca" pod "d72c2c14-562e-4616-8c28-c7da12b45d22" (UID: "d72c2c14-562e-4616-8c28-c7da12b45d22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:02.325410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.325378    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz" pod "d72c2c14-562e-4616-8c28-c7da12b45d22" (UID: "d72c2c14-562e-4616-8c28-c7da12b45d22"). InnerVolumeSpecName "kube-api-access-tf9vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:02.425233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.425197    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9vz\" (UniqueName: \"kubernetes.io/projected/d72c2c14-562e-4616-8c28-c7da12b45d22-kube-api-access-tf9vz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:02.425233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.425227    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72c2c14-562e-4616-8c28-c7da12b45d22-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:02.605010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.604929    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:14:02.605010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.604984    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:14:02.618022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.617989    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc" event={"ID":"d72c2c14-562e-4616-8c28-c7da12b45d22","Type":"ContainerDied","Data":"607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98"}
Oct 30 07:14:02.618196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.618026    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"
Oct 30 07:14:02.618196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.618039    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607fd0cde4291da747905dcb46b3a0a2a4e78eab1a633ccf90145ebf5e24db98"
Oct 30 07:14:02.900208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:02.900172    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:03.034860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:03.034820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:03.605317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:03.604793    6738 patch_prober.go:29] interesting pod/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:14:03.605317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:03.604898    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 07:14:04.050621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:04.050581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:04.617098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:04.617053    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" podUID="9be70749-a046-4fd3-9f79-80069fc179e7" containerName="payload-pre-processing" containerID="cri-o://4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645" gracePeriod=25
Oct 30 07:14:04.627364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:04.627330    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:14:04.641847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:04.641799    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:14:04.934071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:04.934038    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:14:05.028614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.028584    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\") pod \"9be70749-a046-4fd3-9f79-80069fc179e7\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") "
Oct 30 07:14:05.028902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.028887    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\" (UniqueName: \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\") pod \"9be70749-a046-4fd3-9f79-80069fc179e7\" (UID: \"9be70749-a046-4fd3-9f79-80069fc179e7\") "
Oct 30 07:14:05.029079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.029052    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume" pod "9be70749-a046-4fd3-9f79-80069fc179e7" (UID: "9be70749-a046-4fd3-9f79-80069fc179e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:05.029558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.029536    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be70749-a046-4fd3-9f79-80069fc179e7-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:05.030958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.030928    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms" pod "9be70749-a046-4fd3-9f79-80069fc179e7" (UID: "9be70749-a046-4fd3-9f79-80069fc179e7"). InnerVolumeSpecName "kube-api-access-h4hms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:05.130848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.130812    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hms\" (UniqueName: \"kubernetes.io/projected/9be70749-a046-4fd3-9f79-80069fc179e7-kube-api-access-h4hms\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:05.733542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.733508    6738 generic.go:356] "Generic (PLEG): container finished" podID="9be70749-a046-4fd3-9f79-80069fc179e7" containerID="4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645" exitCode=0
Oct 30 07:14:05.734024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.733559    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"
Oct 30 07:14:05.734024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.733591    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" event={"ID":"9be70749-a046-4fd3-9f79-80069fc179e7","Type":"ContainerDied","Data":"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"}
Oct 30 07:14:05.734024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.733647    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k" event={"ID":"9be70749-a046-4fd3-9f79-80069fc179e7","Type":"ContainerDied","Data":"7132af4f46c7b696cbc93772a9e2ac5b4c0cec8ab9ec2fa07abd91d3d9e5a495"}
Oct 30 07:14:05.734024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.733673    6738 scope.go:122] "RemoveContainer" containerID="4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"
Oct 30 07:14:05.745585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.745565    6738 scope.go:122] "RemoveContainer" containerID="4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"
Oct 30 07:14:05.745892 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:05.745857    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645\": container with ID starting with 4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645 not found: ID does not exist" containerID="4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"
Oct 30 07:14:05.745991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.745889    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645"} err="failed to get container status \"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645\": rpc error: code = NotFound desc = could not find container \"4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645\": container with ID starting with 4beade6ebd1c945b741bff805f46508966db6f5a558028d7442da637cc599645 not found: ID does not exist"
Oct 30 07:14:05.789462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.789429    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:14:05.801969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.801937    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a163b2-858447df7f-4lc7k"]
Oct 30 07:14:05.901336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:05.901289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:06.065374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:06.065241    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:06.205592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:06.205558    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:07.519545 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:07.519508    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:14:07.629207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.629171    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:07.695994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.695955    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be70749-a046-4fd3-9f79-80069fc179e7" path="/var/lib/kubelet/pods/9be70749-a046-4fd3-9f79-80069fc179e7/volumes"
Oct 30 07:14:07.756427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.756391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:07.804660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.804563    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:14:07.805420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805398    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d72c2c14-562e-4616-8c28-c7da12b45d22" containerName="revoke-keys"
Oct 30 07:14:07.805420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805420    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72c2c14-562e-4616-8c28-c7da12b45d22" containerName="revoke-keys"
Oct 30 07:14:07.805601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805482    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9be70749-a046-4fd3-9f79-80069fc179e7" containerName="payload-pre-processing"
Oct 30 07:14:07.805601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805490    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be70749-a046-4fd3-9f79-80069fc179e7" containerName="payload-pre-processing"
Oct 30 07:14:07.805709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805650    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be70749-a046-4fd3-9f79-80069fc179e7" containerName="payload-pre-processing"
Oct 30 07:14:07.805709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.805673    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72c2c14-562e-4616-8c28-c7da12b45d22" containerName="revoke-keys"
Oct 30 07:14:07.809259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.809237    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:07.826868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.826834    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:14:07.833453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.833422    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a273fd\""
Oct 30 07:14:07.833453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.833423    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a273fd-dockercfg-jrkg8\""
Oct 30 07:14:07.896508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.887647    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zrk\" (UniqueName: \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:07.896508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:07.887746    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.125347    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.131980    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zrk\" (UniqueName: \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.132100    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.134552    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.136191    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.136362    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.208539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.196051    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zrk\" (UniqueName: \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.245360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.243637    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblgg\" (UniqueName: \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.245360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.243838    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.345354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.345279    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblgg\" (UniqueName: \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.345571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.345512    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.346206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.346185    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.369681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.369645    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblgg\" (UniqueName: \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\") pod \"payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.424620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.424583    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:08.542646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.542600    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:08.623404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.623372    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:08.799645 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:08.799593    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5050b7_59ff_419c_baa1_36a74f6e7a46.slice/crio-440e36104097d6d9d5920d1d26547ec7a61062592e32b9e3c7dca763fcb403ef WatchSource:0}: Error finding container 440e36104097d6d9d5920d1d26547ec7a61062592e32b9e3c7dca763fcb403ef: Status 404 returned error can't find the container with id 440e36104097d6d9d5920d1d26547ec7a61062592e32b9e3c7dca763fcb403ef
Oct 30 07:14:08.802502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.802473    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:14:08.868888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:08.868860    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:14:08.874926 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:08.874897    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0552d7cc_b654_4de8_af6f_fd26ac21f86d.slice/crio-0aba556f4a6a60147cfdef36bca7accf74e3451c5c0afc0731997532882744ef WatchSource:0}: Error finding container 0aba556f4a6a60147cfdef36bca7accf74e3451c5c0afc0731997532882744ef: Status 404 returned error can't find the container with id 0aba556f4a6a60147cfdef36bca7accf74e3451c5c0afc0731997532882744ef
Oct 30 07:14:09.032283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.032248    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"]
Oct 30 07:14:09.036580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.036553    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37061ad2-nvn2v"]
Oct 30 07:14:09.336947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.336912    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:14:09.374062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.374027    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:14:09.594478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.594386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:09.719319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.717888    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b968b3-3fa2-4309-b085-ca6a155b363c" path="/var/lib/kubelet/pods/24b968b3-3fa2-4309-b085-ca6a155b363c/volumes"
Oct 30 07:14:09.747322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.734786    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:09.848125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.848036    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" event={"ID":"0552d7cc-b654-4de8-af6f-fd26ac21f86d","Type":"ContainerStarted","Data":"d6d590bb7af3f5e207b3534fc6f138268ab34ef617ca01aed01732731bab40e7"}
Oct 30 07:14:09.848426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.848383    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" event={"ID":"0552d7cc-b654-4de8-af6f-fd26ac21f86d","Type":"ContainerStarted","Data":"0aba556f4a6a60147cfdef36bca7accf74e3451c5c0afc0731997532882744ef"}
Oct 30 07:14:09.850130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.850105    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:09.850989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.850961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" event={"ID":"9b5050b7-59ff-419c-baa1-36a74f6e7a46","Type":"ContainerStarted","Data":"e3896df1634fc606390ac41cdf0d6fa03523ca6a26f48114ca767c36e4045bd4"}
Oct 30 07:14:09.851100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.851009    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" event={"ID":"9b5050b7-59ff-419c-baa1-36a74f6e7a46","Type":"ContainerStarted","Data":"440e36104097d6d9d5920d1d26547ec7a61062592e32b9e3c7dca763fcb403ef"}
Oct 30 07:14:09.854319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.851525    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:09.893696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.893642    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" podStartSLOduration=2.893626856 podStartE2EDuration="2.893626856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:09.891447662 +0000 UTC m=+1010.324211555" watchObservedRunningTime="2026-10-30 07:14:09.893626856 +0000 UTC m=+1010.326390748"
Oct 30 07:14:09.934802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:09.934744    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" podStartSLOduration=2.9347292400000002 podStartE2EDuration="2.93472924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:09.931954195 +0000 UTC m=+1010.364718085" watchObservedRunningTime="2026-10-30 07:14:09.93472924 +0000 UTC m=+1010.367493133"
Oct 30 07:14:11.187799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:11.187760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:12.034575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:12.034541    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"]
Oct 30 07:14:12.046809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:12.046775    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c670d859-q685n"]
Oct 30 07:14:12.563101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:12.563054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:13.146727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:13.146691    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:13.692980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:13.692940    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed617b6-6bc6-4ad1-8f01-9d387208397c" path="/var/lib/kubelet/pods/3ed617b6-6bc6-4ad1-8f01-9d387208397c/volumes"
Oct 30 07:14:14.337792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.337739    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing" containerID="cri-o://617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9" gracePeriod=25
Oct 30 07:14:14.338735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.338708    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"]
Oct 30 07:14:14.343068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.343047    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.360974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.360943    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"]
Oct 30 07:14:14.367855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.367818    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:14:14.367855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.367864    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:14:14.368085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.367933    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:14:14.373418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.373391    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:14:14.374447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.374406    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing" containerID="cri-o://713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd" gracePeriod=25
Oct 30 07:14:14.420967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.420929    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.421144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.420980    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g29n\" (UniqueName: \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.421223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.421196    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.523208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.523174    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.523426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.523353    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.523426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.523377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g29n\" (UniqueName: \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.523872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.523849    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.525901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.525880    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.549254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.549214    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g29n\" (UniqueName: \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\") pod \"maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.658892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.658854    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:14.790357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.790314    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"]
Oct 30 07:14:14.798030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.795985    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:14.817905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.817870    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"]
Oct 30 07:14:14.930009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.929924    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:14.930009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.929979    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m2g\" (UniqueName: \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:14.930235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.930169    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:14.965984 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:14.965952    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b9c30b_e4e8_42b2_8d9f_ed32799c0346.slice/crio-e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947 WatchSource:0}: Error finding container e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947: Status 404 returned error can't find the container with id e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947
Oct 30 07:14:14.967059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:14.967025    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"]
Oct 30 07:14:15.031951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.031920    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.032124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.031977    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m2g\" (UniqueName: \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.032124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.032060    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.032588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.032568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.034459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.034433    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.055644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.055616    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m2g\" (UniqueName: \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\") pod \"maas-api-revoke-sub-6b5bde99-84zb8\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.119804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.119754    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:15.267286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.267190    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:15.734327 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:15.734277    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86388548_e642_49e1_a78b_8e1c4a2aa39a.slice/crio-3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5 WatchSource:0}: Error finding container 3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5: Status 404 returned error can't find the container with id 3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5
Oct 30 07:14:15.735398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:15.735372    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"]
Oct 30 07:14:16.032528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:16.025927    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc" event={"ID":"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346","Type":"ContainerStarted","Data":"ea960f533916189e13e995ba476e30d25245e4310e60419faa9e13a286c2933c"}
Oct 30 07:14:16.032528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:16.025990    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc" event={"ID":"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346","Type":"ContainerStarted","Data":"e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947"}
Oct 30 07:14:16.057626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:16.051944    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8" event={"ID":"86388548-e642-49e1-a78b-8e1c4a2aa39a","Type":"ContainerStarted","Data":"3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5"}
Oct 30 07:14:16.075966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:16.071262    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc" podStartSLOduration=2.07123291 podStartE2EDuration="2.07123291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:16.070416697 +0000 UTC m=+1016.503180592" watchObservedRunningTime="2026-10-30 07:14:16.07123291 +0000 UTC m=+1016.503996803"
Oct 30 07:14:17.062241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:17.062206    6738 generic.go:356] "Generic (PLEG): container finished" podID="e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" containerID="ea960f533916189e13e995ba476e30d25245e4310e60419faa9e13a286c2933c" exitCode=0
Oct 30 07:14:17.062688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:17.062247    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc" event={"ID":"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346","Type":"ContainerDied","Data":"ea960f533916189e13e995ba476e30d25245e4310e60419faa9e13a286c2933c"}
Oct 30 07:14:17.064215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:17.064192    6738 generic.go:356] "Generic (PLEG): container finished" podID="86388548-e642-49e1-a78b-8e1c4a2aa39a" containerID="07608fcca7d0e4da471392b80ed000c5ddd3a4d027db2e05d8e49bc9ccd5e278" exitCode=0
Oct 30 07:14:17.064347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:17.064227    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8" event={"ID":"86388548-e642-49e1-a78b-8e1c4a2aa39a","Type":"ContainerDied","Data":"07608fcca7d0e4da471392b80ed000c5ddd3a4d027db2e05d8e49bc9ccd5e278"}
Oct 30 07:14:17.704741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:17.704703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:18.266116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.266085    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:18.337926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.337857    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:18.350235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.350187    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\" (UniqueName: \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\") pod \"86388548-e642-49e1-a78b-8e1c4a2aa39a\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") "
Oct 30 07:14:18.350690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.350663    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\") pod \"86388548-e642-49e1-a78b-8e1c4a2aa39a\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") "
Oct 30 07:14:18.350827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.350821    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\") pod \"86388548-e642-49e1-a78b-8e1c4a2aa39a\" (UID: \"86388548-e642-49e1-a78b-8e1c4a2aa39a\") "
Oct 30 07:14:18.351392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.351363    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca" pod "86388548-e642-49e1-a78b-8e1c4a2aa39a" (UID: "86388548-e642-49e1-a78b-8e1c4a2aa39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:18.352686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.352664    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g" pod "86388548-e642-49e1-a78b-8e1c4a2aa39a" (UID: "86388548-e642-49e1-a78b-8e1c4a2aa39a"). InnerVolumeSpecName "kube-api-access-n8m2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:18.353141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.353118    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca" pod "86388548-e642-49e1-a78b-8e1c4a2aa39a" (UID: "86388548-e642-49e1-a78b-8e1c4a2aa39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:18.452694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.452654    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\") pod \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") "
Oct 30 07:14:18.452916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.452751    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\" (UniqueName: \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\") pod \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") "
Oct 30 07:14:18.452916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.452875    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\") pod \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\" (UID: \"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346\") "
Oct 30 07:14:18.453246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.453219    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca" pod "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" (UID: "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:18.453421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.453407    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8m2g\" (UniqueName: \"kubernetes.io/projected/86388548-e642-49e1-a78b-8e1c4a2aa39a-kube-api-access-n8m2g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.453462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.453423    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.453462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.453433    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.453462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.453441    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86388548-e642-49e1-a78b-8e1c4a2aa39a-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.454826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.454803    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca" pod "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" (UID: "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:18.454884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.454868    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n" pod "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" (UID: "e8b9c30b-e4e8-42b2-8d9f-ed32799c0346"). InnerVolumeSpecName "kube-api-access-6g29n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:18.555340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.555286    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.555340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.555337    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g29n\" (UniqueName: \"kubernetes.io/projected/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346-kube-api-access-6g29n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:18.672190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.672151    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:18.787175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:18.787129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:19.084110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.084025    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"
Oct 30 07:14:19.084110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.084046    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc" event={"ID":"e8b9c30b-e4e8-42b2-8d9f-ed32799c0346","Type":"ContainerDied","Data":"e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947"}
Oct 30 07:14:19.084110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.084102    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e157a4be23851d3c8bc6b82da894bf7a373fcd06d0afa7137f643fa6e2199947"
Oct 30 07:14:19.085667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.085645    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"
Oct 30 07:14:19.085803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.085683    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8" event={"ID":"86388548-e642-49e1-a78b-8e1c4a2aa39a","Type":"ContainerDied","Data":"3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5"}
Oct 30 07:14:19.085803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.085726    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e53c4563a7638663390082d3cce24eb396795493c99f8e677e5e6289481d1c5"
Oct 30 07:14:19.580992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:19.580951    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:20.896582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:20.896552    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:14:20.903404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:20.903379    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:14:21.477811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:21.477761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:21.616968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:21.616064    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:23.064703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.064669    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"]
Oct 30 07:14:23.071411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.071382    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e27759a8-rhwk2"]
Oct 30 07:14:23.484335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.484279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:23.691781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.691737    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e52924-91bd-42c3-bbad-1c5a8c631665" path="/var/lib/kubelet/pods/b2e52924-91bd-42c3-bbad-1c5a8c631665/volumes"
Oct 30 07:14:23.760608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.760507    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:14:23.760778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:23.760619    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:14:24.793594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:24.793554    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:14:24.829280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:24.829243    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:14:24.939007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:24.938971    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:25.127381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:25.121984    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:25.545376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:25.538604    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:27.336882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:27.336841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:27.456200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:27.456156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:27.606390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:27.606278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:29.166437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:29.166386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:29.429096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:29.429003    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:29.794239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:29.794145    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" podUID="941ef547-a4e4-45d5-87c9-77b49d810231" containerName="payload-processing" containerID="cri-o://8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4" gracePeriod=25
Oct 30 07:14:29.829949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:29.829883    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" podUID="c763f895-9b4b-4e72-97cd-31f40459d067" containerName="payload-pre-processing" containerID="cri-o://f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa" gracePeriod=25
Oct 30 07:14:30.110062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.110029    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:14:30.150775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.150743    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:14:30.207083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.205941    6738 generic.go:356] "Generic (PLEG): container finished" podID="c763f895-9b4b-4e72-97cd-31f40459d067" containerID="f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa" exitCode=0
Oct 30 07:14:30.207083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.206141    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"
Oct 30 07:14:30.207083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.206559    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" event={"ID":"c763f895-9b4b-4e72-97cd-31f40459d067","Type":"ContainerDied","Data":"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"}
Oct 30 07:14:30.207083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.206611    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8" event={"ID":"c763f895-9b4b-4e72-97cd-31f40459d067","Type":"ContainerDied","Data":"30f6102648d2ff3c78c1d0b4cee385a98f6d7daf37281d4b156186eb97209d5c"}
Oct 30 07:14:30.207083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.206640    6738 scope.go:122] "RemoveContainer" containerID="f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"
Oct 30 07:14:30.211420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.210950    6738 generic.go:356] "Generic (PLEG): container finished" podID="941ef547-a4e4-45d5-87c9-77b49d810231" containerID="8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4" exitCode=0
Oct 30 07:14:30.211420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.211094    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" event={"ID":"941ef547-a4e4-45d5-87c9-77b49d810231","Type":"ContainerDied","Data":"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"}
Oct 30 07:14:30.211420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.211147    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r" event={"ID":"941ef547-a4e4-45d5-87c9-77b49d810231","Type":"ContainerDied","Data":"7eabebf8384724144a5dcd92ff7b57d5a6e6b69a45fe318cb6cef30dfbbf1877"}
Oct 30 07:14:30.211420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.211229    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"
Oct 30 07:14:30.217227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.217201    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\") pod \"c763f895-9b4b-4e72-97cd-31f40459d067\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") "
Oct 30 07:14:30.217410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.217320    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\") pod \"941ef547-a4e4-45d5-87c9-77b49d810231\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") "
Oct 30 07:14:30.217410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.217359    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\" (UniqueName: \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\") pod \"941ef547-a4e4-45d5-87c9-77b49d810231\" (UID: \"941ef547-a4e4-45d5-87c9-77b49d810231\") "
Oct 30 07:14:30.217532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.217514    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\" (UniqueName: \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\") pod \"c763f895-9b4b-4e72-97cd-31f40459d067\" (UID: \"c763f895-9b4b-4e72-97cd-31f40459d067\") "
Oct 30 07:14:30.217755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.217722    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume" pod "c763f895-9b4b-4e72-97cd-31f40459d067" (UID: "c763f895-9b4b-4e72-97cd-31f40459d067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:30.218080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.218054    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume" pod "941ef547-a4e4-45d5-87c9-77b49d810231" (UID: "941ef547-a4e4-45d5-87c9-77b49d810231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:30.218471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.218450    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c763f895-9b4b-4e72-97cd-31f40459d067-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:30.218689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.218476    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941ef547-a4e4-45d5-87c9-77b49d810231-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:30.220528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.220506    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv" pod "941ef547-a4e4-45d5-87c9-77b49d810231" (UID: "941ef547-a4e4-45d5-87c9-77b49d810231"). InnerVolumeSpecName "kube-api-access-zqzgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:30.220656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.220639    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq" pod "c763f895-9b4b-4e72-97cd-31f40459d067" (UID: "c763f895-9b4b-4e72-97cd-31f40459d067"). InnerVolumeSpecName "kube-api-access-knqmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:30.222272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.222249    6738 scope.go:122] "RemoveContainer" containerID="f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"
Oct 30 07:14:30.222736 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:30.222675    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa\": container with ID starting with f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa not found: ID does not exist" containerID="f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"
Oct 30 07:14:30.222814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.222745    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa"} err="failed to get container status \"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa\": rpc error: code = NotFound desc = could not find container \"f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa\": container with ID starting with f23c4c7532e75b737228788bed2e98fe94ed15b241392ee779637540e90328fa not found: ID does not exist"
Oct 30 07:14:30.222814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.222764    6738 scope.go:122] "RemoveContainer" containerID="8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"
Oct 30 07:14:30.241523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.241494    6738 scope.go:122] "RemoveContainer" containerID="8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"
Oct 30 07:14:30.241915 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:30.241891    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4\": container with ID starting with 8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4 not found: ID does not exist" containerID="8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"
Oct 30 07:14:30.242000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.241926    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4"} err="failed to get container status \"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4\": rpc error: code = NotFound desc = could not find container \"8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4\": container with ID starting with 8dd862a7df4cd425ca14c66727bf15ecfaa5cdf2cb4b32f5bf095b7c14dc93c4 not found: ID does not exist"
Oct 30 07:14:30.344540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.330133    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzgv\" (UniqueName: \"kubernetes.io/projected/941ef547-a4e4-45d5-87c9-77b49d810231-kube-api-access-zqzgv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:30.344540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.330167    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqmq\" (UniqueName: \"kubernetes.io/projected/c763f895-9b4b-4e72-97cd-31f40459d067-kube-api-access-knqmq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:30.561340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.561265    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:14:30.573517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.573483    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bda71-6fc7b6db55-mhhk8"]
Oct 30 07:14:30.608735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.608640    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:14:30.613625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:30.613593    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bda71-667649498b-bmj4r"]
Oct 30 07:14:31.347891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:31.347854    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:31.691325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:31.691274    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941ef547-a4e4-45d5-87c9-77b49d810231" path="/var/lib/kubelet/pods/941ef547-a4e4-45d5-87c9-77b49d810231/volumes"
Oct 30 07:14:31.691779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:31.691765    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c763f895-9b4b-4e72-97cd-31f40459d067" path="/var/lib/kubelet/pods/c763f895-9b4b-4e72-97cd-31f40459d067/volumes"
Oct 30 07:14:33.151111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:33.151068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:33.759831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:33.759795    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:14:33.759831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:33.759823    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:14:34.672227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.672193    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:14:34.717442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.717403    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:14:34.819510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819463    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\") pod \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") "
Oct 30 07:14:34.819690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819614    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\") pod \"520c5be5-63b7-4335-9ecb-d74145db8677\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") "
Oct 30 07:14:34.819690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819647    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\" (UniqueName: \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\") pod \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\" (UID: \"595b8a9d-3a0c-485a-b26e-ea62d9714e20\") "
Oct 30 07:14:34.819899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819877    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\" (UniqueName: \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\") pod \"520c5be5-63b7-4335-9ecb-d74145db8677\" (UID: \"520c5be5-63b7-4335-9ecb-d74145db8677\") "
Oct 30 07:14:34.819989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819926    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume" pod "595b8a9d-3a0c-485a-b26e-ea62d9714e20" (UID: "595b8a9d-3a0c-485a-b26e-ea62d9714e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:34.819989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.819974    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume" pod "520c5be5-63b7-4335-9ecb-d74145db8677" (UID: "520c5be5-63b7-4335-9ecb-d74145db8677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:34.820528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.820505    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595b8a9d-3a0c-485a-b26e-ea62d9714e20-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:34.820528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.820528    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520c5be5-63b7-4335-9ecb-d74145db8677-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:34.821903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.821870    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645" pod "520c5be5-63b7-4335-9ecb-d74145db8677" (UID: "520c5be5-63b7-4335-9ecb-d74145db8677"). InnerVolumeSpecName "kube-api-access-8v645". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:34.821995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.821904    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9" pod "595b8a9d-3a0c-485a-b26e-ea62d9714e20" (UID: "595b8a9d-3a0c-485a-b26e-ea62d9714e20"). InnerVolumeSpecName "kube-api-access-tzgw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:34.922579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.922535    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v645\" (UniqueName: \"kubernetes.io/projected/520c5be5-63b7-4335-9ecb-d74145db8677-kube-api-access-8v645\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:34.922579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:34.922571    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgw9\" (UniqueName: \"kubernetes.io/projected/595b8a9d-3a0c-485a-b26e-ea62d9714e20-kube-api-access-tzgw9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:35.038446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.038410    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:35.327891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.327797    6738 generic.go:356] "Generic (PLEG): container finished" podID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerID="617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9" exitCode=0
Oct 30 07:14:35.327891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.327863    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"
Oct 30 07:14:35.328125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.327886    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" event={"ID":"595b8a9d-3a0c-485a-b26e-ea62d9714e20","Type":"ContainerDied","Data":"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"}
Oct 30 07:14:35.328125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.327968    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96" event={"ID":"595b8a9d-3a0c-485a-b26e-ea62d9714e20","Type":"ContainerDied","Data":"4667354883bede0c45958cc4659a2af6b60f80194f26657fcfb53c5840dd396c"}
Oct 30 07:14:35.328125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.328001    6738 scope.go:122] "RemoveContainer" containerID="617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"
Oct 30 07:14:35.329589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.329567    6738 generic.go:356] "Generic (PLEG): container finished" podID="520c5be5-63b7-4335-9ecb-d74145db8677" containerID="713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd" exitCode=0
Oct 30 07:14:35.329703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.329635    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"
Oct 30 07:14:35.329746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.329630    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" event={"ID":"520c5be5-63b7-4335-9ecb-d74145db8677","Type":"ContainerDied","Data":"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"}
Oct 30 07:14:35.329792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.329740    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz" event={"ID":"520c5be5-63b7-4335-9ecb-d74145db8677","Type":"ContainerDied","Data":"c4ce67b8c58a6d619542be23c3172578ff794f04d14d320b749ed48f3e731031"}
Oct 30 07:14:35.339598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.339541    6738 scope.go:122] "RemoveContainer" containerID="617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"
Oct 30 07:14:35.339976 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:35.339949    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9\": container with ID starting with 617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9 not found: ID does not exist" containerID="617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"
Oct 30 07:14:35.340074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.339984    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9"} err="failed to get container status \"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9\": rpc error: code = NotFound desc = could not find container \"617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9\": container with ID starting with 617a1d6a88a2c0452488a77c59d69d2d1538e4e1d7001c5f0151f726ef8fc5d9 not found: ID does not exist"
Oct 30 07:14:35.340074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.340008    6738 scope.go:122] "RemoveContainer" containerID="713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"
Oct 30 07:14:35.350794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.350773    6738 scope.go:122] "RemoveContainer" containerID="713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"
Oct 30 07:14:35.351116 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:35.351094    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd\": container with ID starting with 713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd not found: ID does not exist" containerID="713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"
Oct 30 07:14:35.351201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.351127    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd"} err="failed to get container status \"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd\": rpc error: code = NotFound desc = could not find container \"713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd\": container with ID starting with 713ec64a70c40d1a0b98ce03cc2b9857c17a815083d1a57e4671a681694ed9cd not found: ID does not exist"
Oct 30 07:14:35.383398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.383362    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:14:35.395252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.395219    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-lwt96"]
Oct 30 07:14:35.422619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.422566    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:14:35.427589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.427564    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-mc7pz"]
Oct 30 07:14:35.691170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.691139    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" path="/var/lib/kubelet/pods/520c5be5-63b7-4335-9ecb-d74145db8677/volumes"
Oct 30 07:14:35.691655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:35.691641    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" path="/var/lib/kubelet/pods/595b8a9d-3a0c-485a-b26e-ea62d9714e20/volumes"
Oct 30 07:14:36.734373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:36.734342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:37.533062 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:14:37.533006    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:14:37.553861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:37.553811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:37.706406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:37.706368    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:37.766381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:37.747697    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"]
Oct 30 07:14:37.792371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:37.781286    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-cfrmc"]
Oct 30 07:14:37.901139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:37.901104    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:39.691602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:39.691574    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72c2c14-562e-4616-8c28-c7da12b45d22" path="/var/lib/kubelet/pods/d72c2c14-562e-4616-8c28-c7da12b45d22/volumes"
Oct 30 07:14:39.850923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:39.850886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:41.174091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:41.174039    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:41.307318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:41.305958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:41.868579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:41.868538    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:42.091239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:42.091202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:43.726190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:43.726160    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:44.869816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:44.869778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:44.987315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:44.986794    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:45.068315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:45.064360    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"]
Oct 30 07:14:45.078252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:45.078203    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a163b2-bvfj4"]
Oct 30 07:14:45.100668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:45.100629    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:45.548031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:45.547989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:45.701393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:45.701356    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e423ee77-f9b3-417a-9a80-8cb5282b0667" path="/var/lib/kubelet/pods/e423ee77-f9b3-417a-9a80-8cb5282b0667/volumes"
Oct 30 07:14:46.287317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:46.286397    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:46.892315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:46.891482    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:47.364150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:47.364060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:47.601239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:47.601208    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:48.052148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.052111    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:14:48.052973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.052951    6738 cpu_manager.go:414] "containerMap: removing container" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.052983    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053051    6738 cpu_manager.go:414] "containerMap: removing container" podUID="86388548-e642-49e1-a78b-8e1c4a2aa39a" containerName="revoke-keys"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053060    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86388548-e642-49e1-a78b-8e1c4a2aa39a" containerName="revoke-keys"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053089    6738 cpu_manager.go:414] "containerMap: removing container" podUID="941ef547-a4e4-45d5-87c9-77b49d810231" containerName="payload-processing"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053097    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941ef547-a4e4-45d5-87c9-77b49d810231" containerName="payload-processing"
Oct 30 07:14:48.053125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053122    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c763f895-9b4b-4e72-97cd-31f40459d067" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053130    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c763f895-9b4b-4e72-97cd-31f40459d067" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053144    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" containerName="revoke-keys"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053152    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" containerName="revoke-keys"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053172    6738 cpu_manager.go:414] "containerMap: removing container" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053181    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053368    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c763f895-9b4b-4e72-97cd-31f40459d067" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053385    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" containerName="revoke-keys"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053398    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="520c5be5-63b7-4335-9ecb-d74145db8677" containerName="payload-pre-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053411    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="595b8a9d-3a0c-485a-b26e-ea62d9714e20" containerName="payload-processing"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053426    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="86388548-e642-49e1-a78b-8e1c4a2aa39a" containerName="revoke-keys"
Oct 30 07:14:48.053448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.053437    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="941ef547-a4e4-45d5-87c9-77b49d810231" containerName="payload-processing"
Oct 30 07:14:48.057803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.057774    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.087537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.087503    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4e42c8-openshift-default-dockercfg-p8w58\""
Oct 30 07:14:48.090389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.090359    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:14:48.118076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.118039    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:48.195549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195514    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.195726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195647    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.195726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195719    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.195865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195748    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.195865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195837    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmth5\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.195974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.195961    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.196050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.196032    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.196093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.196069    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.196146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.196101    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.278382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.278342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:48.303448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.301740    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.313717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.313674    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.313999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.313978    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.314162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.314126    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.316719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.316683    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.326586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.326556    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.327504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.327478    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.327984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.327966    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.347316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.329002    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmth5\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.347718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.327248    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.347905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.317236    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.348021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.317825    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.348126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.327899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.348219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.318049    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.348386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.328943    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.348532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.323509    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.361321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.361273    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmth5\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.523319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.517105    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\") pod \"e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") " pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:48.688558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:48.688519    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:49.031955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.031929    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:14:49.034183 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:49.034154    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecbecbb_2144_45db_8df9_47357b2dbf9f.slice/crio-ed18d3aed87deba7157a7998b98342121284198a7d79124b4c14abe67426f498 WatchSource:0}: Error finding container ed18d3aed87deba7157a7998b98342121284198a7d79124b4c14abe67426f498: Status 404 returned error can't find the container with id ed18d3aed87deba7157a7998b98342121284198a7d79124b4c14abe67426f498
Oct 30 07:14:49.037140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.037099    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:49.037259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.037179    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:49.037259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.037223    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:49.714578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.714538    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:49.744655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.744600    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerStarted","Data":"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"}
Oct 30 07:14:49.744655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.744658    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerStarted","Data":"ed18d3aed87deba7157a7998b98342121284198a7d79124b4c14abe67426f498"}
Oct 30 07:14:49.815529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.815462    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podStartSLOduration=1.815438951 podStartE2EDuration="1.815438951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:49.808999519 +0000 UTC m=+1050.241763414" watchObservedRunningTime="2026-10-30 07:14:49.815438951 +0000 UTC m=+1050.248202847"
Oct 30 07:14:49.860087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:49.860021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:50.633330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.633256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:50.688753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.688715    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:50.694862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.694830    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:50.755329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.755259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:50.756412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.756381    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:14:50.769602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.769553    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:50.903275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.903179    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:14:50.908767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.908736    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:50.927982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.927591    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a6a14e07\""
Oct 30 07:14:50.967349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.952972    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:14:50.967349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:50.953368    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a6a14e07-dockercfg-b4ff6\""
Oct 30 07:14:51.028368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.027686    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjsg\" (UniqueName: \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.028368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.027854    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.187406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.135016    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjsg\" (UniqueName: \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.187406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.135304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.187406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.139824    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.254320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.232234    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjsg\" (UniqueName: \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\") pod \"payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.277808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.277766    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:51.648065 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:51.648037    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0cee4d_8344_44bd_a2e8_a5fbdc2c1594.slice/crio-f64bc302f2cc229faff2fd496f70471ad06be2407162a71cd15300314ac829f5 WatchSource:0}: Error finding container f64bc302f2cc229faff2fd496f70471ad06be2407162a71cd15300314ac829f5: Status 404 returned error can't find the container with id f64bc302f2cc229faff2fd496f70471ad06be2407162a71cd15300314ac829f5
Oct 30 07:14:51.651630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:51.651595    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:14:52.055379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:52.038510    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" event={"ID":"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594","Type":"ContainerStarted","Data":"f64bc302f2cc229faff2fd496f70471ad06be2407162a71cd15300314ac829f5"}
Oct 30 07:14:52.235212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:52.235099    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:52.363362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:52.359217    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:52.567507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:52.567461    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:52.690718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:52.690679    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:53.087583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.085713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" event={"ID":"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594","Type":"ContainerStarted","Data":"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"}
Oct 30 07:14:53.087583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.085797    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:14:53.127731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.127261    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" podStartSLOduration=3.127244227 podStartE2EDuration="3.127244227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:53.126766571 +0000 UTC m=+1053.559530465" watchObservedRunningTime="2026-10-30 07:14:53.127244227 +0000 UTC m=+1053.560008317"
Oct 30 07:14:53.268200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.268162    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"]
Oct 30 07:14:53.309321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.308896    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.327343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.325001    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:14:53.327343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.325718    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:14:53.327343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.325916    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:14:53.327343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.325954    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:14:53.330726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.329768    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"]
Oct 30 07:14:53.459616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.459572    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.459799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.459746    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbrb\" (UniqueName: \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.459881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.459799    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.561086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.561046    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.561268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.561251    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.561427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.561413    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbrb\" (UniqueName: \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.562073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.562044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.563631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.563603    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.585198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.585154    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbrb\" (UniqueName: \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\") pod \"maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:53.712184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:53.712148    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:54.040990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:54.040955    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"]
Oct 30 07:14:54.041514 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:14:54.041481    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55804b2b_4b87_4ee2_b7b6_554297b61d4e.slice/crio-675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c WatchSource:0}: Error finding container 675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c: Status 404 returned error can't find the container with id 675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c
Oct 30 07:14:54.094375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:54.094339    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds" event={"ID":"55804b2b-4b87-4ee2-b7b6-554297b61d4e","Type":"ContainerStarted","Data":"675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c"}
Oct 30 07:14:54.463626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:54.463590    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:54.581049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:54.580994    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:54.740658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:54.740623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:55.118711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.118679    6738 generic.go:356] "Generic (PLEG): container finished" podID="7495b90f-0807-44c4-a023-7dfe4107408b" containerID="a3bee4efcf6e7664063ba3be38ecb7a64d0a2fd231f2e93af252daaf868ed781" exitCode=0
Oct 30 07:14:55.119213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.118936    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerDied","Data":"a3bee4efcf6e7664063ba3be38ecb7a64d0a2fd231f2e93af252daaf868ed781"}
Oct 30 07:14:55.120110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.119482    6738 scope.go:122] "RemoveContainer" containerID="a3bee4efcf6e7664063ba3be38ecb7a64d0a2fd231f2e93af252daaf868ed781"
Oct 30 07:14:55.122074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.122042    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:55.122191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.122120    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:55.122191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.122159    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:14:55.122312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.122266    6738 generic.go:356] "Generic (PLEG): container finished" podID="55804b2b-4b87-4ee2-b7b6-554297b61d4e" containerID="792b9e02bc587c950e59aa138f8ce2c816c08aef9dd1fc19ac6cf53e956e45d4" exitCode=0
Oct 30 07:14:55.122355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:55.122320    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds" event={"ID":"55804b2b-4b87-4ee2-b7b6-554297b61d4e","Type":"ContainerDied","Data":"792b9e02bc587c950e59aa138f8ce2c816c08aef9dd1fc19ac6cf53e956e45d4"}
Oct 30 07:14:56.133100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.133061    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerStarted","Data":"3daff7c6a342345b4cdd4aaa6f11f6a6d1be586be893f91a18102c3f19469b88"}
Oct 30 07:14:56.335081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.335055    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:56.436700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.436599    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\") pod \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") "
Oct 30 07:14:56.436700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.436675    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\") pod \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") "
Oct 30 07:14:56.436976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.436852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\" (UniqueName: \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\") pod \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\" (UID: \"55804b2b-4b87-4ee2-b7b6-554297b61d4e\") "
Oct 30 07:14:56.437140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.437105    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca" pod "55804b2b-4b87-4ee2-b7b6-554297b61d4e" (UID: "55804b2b-4b87-4ee2-b7b6-554297b61d4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:56.437573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.437546    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:56.439375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.439346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb" pod "55804b2b-4b87-4ee2-b7b6-554297b61d4e" (UID: "55804b2b-4b87-4ee2-b7b6-554297b61d4e"). InnerVolumeSpecName "kube-api-access-5dbrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:56.439482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.439458    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca" pod "55804b2b-4b87-4ee2-b7b6-554297b61d4e" (UID: "55804b2b-4b87-4ee2-b7b6-554297b61d4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:56.539561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.539525    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbrb\" (UniqueName: \"kubernetes.io/projected/55804b2b-4b87-4ee2-b7b6-554297b61d4e-kube-api-access-5dbrb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:56.539561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.539556    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55804b2b-4b87-4ee2-b7b6-554297b61d4e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:14:56.822912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.822808    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:56.910179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.910145    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:14:56.910623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.910560    6738 patch_prober.go:29] interesting pod/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 07:14:56.910970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:56.910937    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 07:14:57.007711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.007678    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:14:57.036367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.033162    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:14:57.084370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.083957    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:57.156498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.156449    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds" event={"ID":"55804b2b-4b87-4ee2-b7b6-554297b61d4e","Type":"ContainerDied","Data":"675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c"}
Oct 30 07:14:57.157013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.156552    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="675b87514dd24147e0d410da803d9d66140f6b07cce63bbf3f3b23e9f201101c"
Oct 30 07:14:57.157013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.156649    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"
Oct 30 07:14:57.232437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.230521    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:57.469283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.469237    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:57.922321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.917754    6738 patch_prober.go:29] interesting pod/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 07:14:57.922321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:57.917813    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 07:14:58.907957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:58.907916    6738 patch_prober.go:29] interesting pod/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 07:14:58.908411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:58.907972    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 07:14:58.996328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:58.996276    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:14:59.510422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:59.510385    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:14:59.907948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:59.907914    6738 patch_prober.go:29] interesting pod/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 07:14:59.908133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:14:59.907969    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 07:15:00.157151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.157114    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:00.157897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.157879    6738 cpu_manager.go:414] "containerMap: removing container" podUID="55804b2b-4b87-4ee2-b7b6-554297b61d4e" containerName="revoke-keys"
Oct 30 07:15:00.157986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.157899    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55804b2b-4b87-4ee2-b7b6-554297b61d4e" containerName="revoke-keys"
Oct 30 07:15:00.158044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.158035    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="55804b2b-4b87-4ee2-b7b6-554297b61d4e" containerName="revoke-keys"
Oct 30 07:15:00.161826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.161799    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.162494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.162473    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:00.176326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.167445    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:00.180329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.168092    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.192932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.192900    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:15:00.205417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.193441    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:15:00.205417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.193597    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:15:00.211678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.211645    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:00.212957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.212921    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.216740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.216707    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:00.216905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.216884    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.217119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.217094    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:00.224342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.224272    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:00.230947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.230904    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:00.235919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.235887    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.237321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.237277    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.237542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.237416    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:00.245423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.245397    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:00.281353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.279643    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.307345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.306847    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:00.311493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.311465    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pmx\" (UniqueName: \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.311684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.311669    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.311761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.311751    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km52\" (UniqueName: \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.311995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.311977    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llg7t\" (UniqueName: \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.312219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.312201    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.312497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.312475    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.312586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.312575    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vw8\" (UniqueName: \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.312699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.312672    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.316231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.316208    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:00.325571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.324610    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:00.332497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.332468    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:00.414219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414126    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2946r\" (UniqueName: \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.414219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414195    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pmx\" (UniqueName: \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.414443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414264    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.414443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414322    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km52\" (UniqueName: \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.414443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414423    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.414554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414452    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chwz\" (UniqueName: \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.414554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414475    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llg7t\" (UniqueName: \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.414710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414693    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.414816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414797    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.414920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414900    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vw8\" (UniqueName: \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.414987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.414946    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.415047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415001    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.415047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415012    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.415047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415029    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.415210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415157    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dxq\" (UniqueName: \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.415475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415455    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.415592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415536    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.415646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.415597    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.438885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.438854    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pmx\" (UniqueName: \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\") pod \"maas-api-key-cleanup-29889075-hzf85\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.439096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.438858    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km52\" (UniqueName: \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.439096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.439018    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llg7t\" (UniqueName: \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\") pod \"maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.439228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.439136    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vw8\" (UniqueName: \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\") pod \"maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.506131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.506095    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:00.516488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.516456    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.516694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.516675    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.517435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517409    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.517435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517423    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dxq\" (UniqueName: \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.517435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517281    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.517612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517572    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2946r\" (UniqueName: \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.517683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517665    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.517757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.517706    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chwz\" (UniqueName: \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.518469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.518445    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.541365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.541321    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2946r\" (UniqueName: \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\") pod \"maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.541563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.541378    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dxq\" (UniqueName: \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.541722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.541700    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chwz\" (UniqueName: \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.548500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.548468    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:00.617349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.615544    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:00.631839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.629233    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:00.635915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.635836    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:00.644648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.644174    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:00.653810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.653427    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:00.904424 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:00.903746    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b10586_f302_4ef0_9b02_36d388d0c6e5.slice/crio-b3035b38293851732cebee027bac937542f1b17dacdfd0d8a89809e775feaec6 WatchSource:0}: Error finding container b3035b38293851732cebee027bac937542f1b17dacdfd0d8a89809e775feaec6: Status 404 returned error can't find the container with id b3035b38293851732cebee027bac937542f1b17dacdfd0d8a89809e775feaec6
Oct 30 07:15:00.908243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.908207    6738 patch_prober.go:29] interesting pod/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 07:15:00.908670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.908265    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 07:15:00.913025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:00.912888    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:01.008203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.008123    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:01.227596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.227157    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:01.255058 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:01.246715    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda289e534_9b3f_4b4d_ad4c_7fc9dd163688.slice/crio-01d16adb6890bc726bad5361234453c1458f41b88e414f37cc22b915d5cd8187 WatchSource:0}: Error finding container 01d16adb6890bc726bad5361234453c1458f41b88e414f37cc22b915d5cd8187: Status 404 returned error can't find the container with id 01d16adb6890bc726bad5361234453c1458f41b88e414f37cc22b915d5cd8187
Oct 30 07:15:01.356224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.355928    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:01.368427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.363236    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:01.409325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.406565    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:01.409325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.407941    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerStarted","Data":"b3035b38293851732cebee027bac937542f1b17dacdfd0d8a89809e775feaec6"}
Oct 30 07:15:01.433287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.432436    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:01.435092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.435057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerStarted","Data":"01d16adb6890bc726bad5361234453c1458f41b88e414f37cc22b915d5cd8187"}
Oct 30 07:15:01.523953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.523918    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:01.530584 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:01.530556    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce38fb1d_6fd8_46b4_911e_046c062e1a52.slice/crio-36fa7144021062611dd613aed9a76118822d8aa81b7ebed72fe84245abd020c7 WatchSource:0}: Error finding container 36fa7144021062611dd613aed9a76118822d8aa81b7ebed72fe84245abd020c7: Status 404 returned error can't find the container with id 36fa7144021062611dd613aed9a76118822d8aa81b7ebed72fe84245abd020c7
Oct 30 07:15:01.591794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.591759    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:01.603282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.603248    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:01.931003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.930974    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:15:01.941114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:01.941074    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:15:02.008870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.008809    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" podUID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" containerName="payload-processing" containerID="cri-o://d6d590bb7af3f5e207b3534fc6f138268ab34ef617ca01aed01732731bab40e7" gracePeriod=25
Oct 30 07:15:02.033629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.033502    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" podUID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" containerName="payload-pre-processing" containerID="cri-o://e3896df1634fc606390ac41cdf0d6fa03523ca6a26f48114ca767c36e4045bd4" gracePeriod=25
Oct 30 07:15:02.516649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.516427    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerStarted","Data":"4de0734818158941b37a1cd84df64a34efa68fa641602e1e98dfe2c47f1dc4c0"}
Oct 30 07:15:02.516649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.516492    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerStarted","Data":"36fa7144021062611dd613aed9a76118822d8aa81b7ebed72fe84245abd020c7"}
Oct 30 07:15:02.519239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.519216    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerStarted","Data":"78ffd02833add3e7ddcf6fca3651acb83530b901587ab45ac9d8edf9fbddca50"}
Oct 30 07:15:02.521882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.521855    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerStarted","Data":"3c3d8b64ff408305dce3a99ad1ac616a4837a0697462f8b89e741b7d8a456c8b"}
Oct 30 07:15:02.531621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.525116    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerStarted","Data":"b552b37eb831a3979f573ab9912b3d4580438d2fb9441ba7db5d8c7b9a92fc68"}
Oct 30 07:15:02.531621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.525169    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerStarted","Data":"9640aa3dcf048e49289777b4f522c79b5f737c7cb46962f8c7e1d52aa8349782"}
Oct 30 07:15:02.531621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.529412    6738 generic.go:356] "Generic (PLEG): container finished" podID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" containerID="d6d590bb7af3f5e207b3534fc6f138268ab34ef617ca01aed01732731bab40e7" exitCode=0
Oct 30 07:15:02.531621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.529510    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" event={"ID":"0552d7cc-b654-4de8-af6f-fd26ac21f86d","Type":"ContainerDied","Data":"d6d590bb7af3f5e207b3534fc6f138268ab34ef617ca01aed01732731bab40e7"}
Oct 30 07:15:02.534220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.534194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerStarted","Data":"d18d905ae057c1c43dac1e09b34ada8a7af29670ba6438b4694596241b6dc5d0"}
Oct 30 07:15:02.534343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.534241    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerStarted","Data":"a941dc9d97c0ba215a26b900338e98bc88cd96dd96f879f497d513685672ec0e"}
Oct 30 07:15:02.537023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.536998    6738 generic.go:356] "Generic (PLEG): container finished" podID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" containerID="e3896df1634fc606390ac41cdf0d6fa03523ca6a26f48114ca767c36e4045bd4" exitCode=0
Oct 30 07:15:02.537130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.537065    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" event={"ID":"9b5050b7-59ff-419c-baa1-36a74f6e7a46","Type":"ContainerDied","Data":"e3896df1634fc606390ac41cdf0d6fa03523ca6a26f48114ca767c36e4045bd4"}
Oct 30 07:15:02.539489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.539461    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerStarted","Data":"9731de5e58c2aff3bbd4a922f26c8e7eafefb0c0d989ca226450ee62794339e7"}
Oct 30 07:15:02.539603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.539512    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerStarted","Data":"df90cbcf52af070111876cc2bbe8492be3895f60aacec7e36a28c1d7c345f764"}
Oct 30 07:15:02.541998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.541947    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerDied","Data":"e36348f09babc22d28f0e5a40b0704dbb3ec5b4acd9df33070ae7d9ddd1834c1"}
Oct 30 07:15:02.542204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.542187    6738 scope.go:122] "RemoveContainer" containerID="e36348f09babc22d28f0e5a40b0704dbb3ec5b4acd9df33070ae7d9ddd1834c1"
Oct 30 07:15:02.542374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.542353    6738 generic.go:356] "Generic (PLEG): container finished" podID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerID="e36348f09babc22d28f0e5a40b0704dbb3ec5b4acd9df33070ae7d9ddd1834c1" exitCode=60
Oct 30 07:15:02.574713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.574527    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" podStartSLOduration=2.57450793 podStartE2EDuration="2.57450793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.567372563 +0000 UTC m=+1063.000136457" watchObservedRunningTime="2026-10-30 07:15:02.57450793 +0000 UTC m=+1063.007271824"
Oct 30 07:15:02.615717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.615653    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" podStartSLOduration=2.6156310830000002 podStartE2EDuration="2.615631083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.609600151 +0000 UTC m=+1063.042364043" watchObservedRunningTime="2026-10-30 07:15:02.615631083 +0000 UTC m=+1063.048394977"
Oct 30 07:15:02.652571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.652521    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" podStartSLOduration=2.652507184 podStartE2EDuration="2.652507184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.649756673 +0000 UTC m=+1063.082520569" watchObservedRunningTime="2026-10-30 07:15:02.652507184 +0000 UTC m=+1063.085271108"
Oct 30 07:15:02.694977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.694916    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" podStartSLOduration=2.694901378 podStartE2EDuration="2.694901378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.691441506 +0000 UTC m=+1063.124205409" watchObservedRunningTime="2026-10-30 07:15:02.694901378 +0000 UTC m=+1063.127665270"
Oct 30 07:15:02.759128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.758311    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" podStartSLOduration=2.7582781929999998 podStartE2EDuration="2.758278193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.731470603 +0000 UTC m=+1063.164234507" watchObservedRunningTime="2026-10-30 07:15:02.758278193 +0000 UTC m=+1063.191042087"
Oct 30 07:15:02.767628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.767598    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:15:02.777359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.777318    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:15:02.802187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802152    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\") pod \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") "
Oct 30 07:15:02.802369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802262    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\" (UniqueName: \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\") pod \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\" (UID: \"9b5050b7-59ff-419c-baa1-36a74f6e7a46\") "
Oct 30 07:15:02.802369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802357    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\") pod \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") "
Oct 30 07:15:02.802509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802391    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\" (UniqueName: \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\") pod \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\" (UID: \"0552d7cc-b654-4de8-af6f-fd26ac21f86d\") "
Oct 30 07:15:02.802627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802602    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume" pod "9b5050b7-59ff-419c-baa1-36a74f6e7a46" (UID: "9b5050b7-59ff-419c-baa1-36a74f6e7a46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:02.803032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.802994    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5050b7-59ff-419c-baa1-36a74f6e7a46-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:02.803263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.803238    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume" pod "0552d7cc-b654-4de8-af6f-fd26ac21f86d" (UID: "0552d7cc-b654-4de8-af6f-fd26ac21f86d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:02.821611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.821515    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" podStartSLOduration=2.821498935 podStartE2EDuration="2.821498935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.81748779 +0000 UTC m=+1063.250251684" watchObservedRunningTime="2026-10-30 07:15:02.821498935 +0000 UTC m=+1063.254262831"
Oct 30 07:15:02.877563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.877017    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk" pod "9b5050b7-59ff-419c-baa1-36a74f6e7a46" (UID: "9b5050b7-59ff-419c-baa1-36a74f6e7a46"). InnerVolumeSpecName "kube-api-access-l4zrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:02.891328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.889059    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg" pod "0552d7cc-b654-4de8-af6f-fd26ac21f86d" (UID: "0552d7cc-b654-4de8-af6f-fd26ac21f86d"). InnerVolumeSpecName "kube-api-access-gblgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:02.927367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.904502    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0552d7cc-b654-4de8-af6f-fd26ac21f86d-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:02.927367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.904537    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblgg\" (UniqueName: \"kubernetes.io/projected/0552d7cc-b654-4de8-af6f-fd26ac21f86d-kube-api-access-gblgg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:02.927367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:02.904553    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zrk\" (UniqueName: \"kubernetes.io/projected/9b5050b7-59ff-419c-baa1-36a74f6e7a46-kube-api-access-l4zrk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:03.224824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.224775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:03.553867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.553832    6738 generic.go:356] "Generic (PLEG): container finished" podID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerID="4de0734818158941b37a1cd84df64a34efa68fa641602e1e98dfe2c47f1dc4c0" exitCode=60
Oct 30 07:15:03.554054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.553918    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerDied","Data":"4de0734818158941b37a1cd84df64a34efa68fa641602e1e98dfe2c47f1dc4c0"}
Oct 30 07:15:03.554231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.554216    6738 scope.go:122] "RemoveContainer" containerID="4de0734818158941b37a1cd84df64a34efa68fa641602e1e98dfe2c47f1dc4c0"
Oct 30 07:15:03.555739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.555717    6738 generic.go:356] "Generic (PLEG): container finished" podID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerID="78ffd02833add3e7ddcf6fca3651acb83530b901587ab45ac9d8edf9fbddca50" exitCode=60
Oct 30 07:15:03.555866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.555802    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerDied","Data":"78ffd02833add3e7ddcf6fca3651acb83530b901587ab45ac9d8edf9fbddca50"}
Oct 30 07:15:03.556101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.556051    6738 scope.go:122] "RemoveContainer" containerID="78ffd02833add3e7ddcf6fca3651acb83530b901587ab45ac9d8edf9fbddca50"
Oct 30 07:15:03.557755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.557732    6738 generic.go:356] "Generic (PLEG): container finished" podID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerID="f99234d11b53c0d0e28ce10a142fce4c20e7914bb1ba10b4912e497853a171cf" exitCode=60
Oct 30 07:15:03.557876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.557823    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerDied","Data":"f99234d11b53c0d0e28ce10a142fce4c20e7914bb1ba10b4912e497853a171cf"}
Oct 30 07:15:03.558075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.558051    6738 scope.go:122] "RemoveContainer" containerID="f99234d11b53c0d0e28ce10a142fce4c20e7914bb1ba10b4912e497853a171cf"
Oct 30 07:15:03.560027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.560005    6738 generic.go:356] "Generic (PLEG): container finished" podID="b0ac0407-ad88-4786-8165-3128f9465b55" containerID="b552b37eb831a3979f573ab9912b3d4580438d2fb9441ba7db5d8c7b9a92fc68" exitCode=60
Oct 30 07:15:03.560131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.560115    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerDied","Data":"b552b37eb831a3979f573ab9912b3d4580438d2fb9441ba7db5d8c7b9a92fc68"}
Oct 30 07:15:03.560503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.560483    6738 scope.go:122] "RemoveContainer" containerID="b552b37eb831a3979f573ab9912b3d4580438d2fb9441ba7db5d8c7b9a92fc68"
Oct 30 07:15:03.561740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.561722    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6" event={"ID":"0552d7cc-b654-4de8-af6f-fd26ac21f86d","Type":"ContainerDied","Data":"0aba556f4a6a60147cfdef36bca7accf74e3451c5c0afc0731997532882744ef"}
Oct 30 07:15:03.561821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.561762    6738 scope.go:122] "RemoveContainer" containerID="d6d590bb7af3f5e207b3534fc6f138268ab34ef617ca01aed01732731bab40e7"
Oct 30 07:15:03.561873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.561851    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"
Oct 30 07:15:03.564267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.564242    6738 generic.go:356] "Generic (PLEG): container finished" podID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerID="d18d905ae057c1c43dac1e09b34ada8a7af29670ba6438b4694596241b6dc5d0" exitCode=60
Oct 30 07:15:03.564524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.564504    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerDied","Data":"d18d905ae057c1c43dac1e09b34ada8a7af29670ba6438b4694596241b6dc5d0"}
Oct 30 07:15:03.564742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.564726    6738 scope.go:122] "RemoveContainer" containerID="d18d905ae057c1c43dac1e09b34ada8a7af29670ba6438b4694596241b6dc5d0"
Oct 30 07:15:03.566400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.566380    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"
Oct 30 07:15:03.566507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.566388    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7" event={"ID":"9b5050b7-59ff-419c-baa1-36a74f6e7a46","Type":"ContainerDied","Data":"440e36104097d6d9d5920d1d26547ec7a61062592e32b9e3c7dca763fcb403ef"}
Oct 30 07:15:03.570396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.570364    6738 generic.go:356] "Generic (PLEG): container finished" podID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerID="9731de5e58c2aff3bbd4a922f26c8e7eafefb0c0d989ca226450ee62794339e7" exitCode=60
Oct 30 07:15:03.570705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.570681    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerDied","Data":"9731de5e58c2aff3bbd4a922f26c8e7eafefb0c0d989ca226450ee62794339e7"}
Oct 30 07:15:03.570970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.570837    6738 scope.go:122] "RemoveContainer" containerID="9731de5e58c2aff3bbd4a922f26c8e7eafefb0c0d989ca226450ee62794339e7"
Oct 30 07:15:03.574771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.574741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerStarted","Data":"4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"}
Oct 30 07:15:03.580099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.580030    6738 scope.go:122] "RemoveContainer" containerID="e3896df1634fc606390ac41cdf0d6fa03523ca6a26f48114ca767c36e4045bd4"
Oct 30 07:15:03.593525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.593496    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:15:03.628796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.628767    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:15:03.747304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.747257    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:03.765125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.765087    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:15:03.773887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.773856    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a273fd-68dc9468d6-qnqx6"]
Oct 30 07:15:03.805986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.805925    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" podStartSLOduration=3.805907242 podStartE2EDuration="3.805907242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:03.801834827 +0000 UTC m=+1064.234598717" watchObservedRunningTime="2026-10-30 07:15:03.805907242 +0000 UTC m=+1064.238671136"
Oct 30 07:15:03.896107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.896038    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:03.913715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.913681    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:15:03.956165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:03.956053    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-84bd8b4967-67zf7"]
Oct 30 07:15:04.679074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.677814    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerStarted","Data":"f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"}
Oct 30 07:15:04.704691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.702626    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerStarted","Data":"af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"}
Oct 30 07:15:04.804377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.802752    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerStarted","Data":"9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"}
Oct 30 07:15:04.890322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.890263    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerStarted","Data":"cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"}
Oct 30 07:15:04.894383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.890787    6738 scope.go:122] "RemoveContainer" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"
Oct 30 07:15:04.894383 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:04.891137    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln_odh-ai-gateway-infra(b0ac0407-ad88-4786-8165-3128f9465b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" podUID="b0ac0407-ad88-4786-8165-3128f9465b55"
Oct 30 07:15:04.898361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.897595    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerStarted","Data":"1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"}
Oct 30 07:15:04.909422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.906577    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerStarted","Data":"118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"}
Oct 30 07:15:04.909718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.909690    6738 generic.go:356] "Generic (PLEG): container finished" podID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerID="4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8" exitCode=60
Oct 30 07:15:04.910501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.909874    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerDied","Data":"4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"}
Oct 30 07:15:04.910765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.910405    6738 scope.go:122] "RemoveContainer" containerID="4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"
Oct 30 07:15:04.911385 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:04.911362    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt_odh-ai-gateway-infra(e5b10586-f302-4ef0-9b02-36d388d0c6e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5"
Oct 30 07:15:04.911781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:04.911497    6738 scope.go:122] "RemoveContainer" containerID="e36348f09babc22d28f0e5a40b0704dbb3ec5b4acd9df33070ae7d9ddd1834c1"
Oct 30 07:15:05.144013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.141874    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:15:05.692127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.692095    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" path="/var/lib/kubelet/pods/0552d7cc-b654-4de8-af6f-fd26ac21f86d/volumes"
Oct 30 07:15:05.692639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.692623    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" path="/var/lib/kubelet/pods/9b5050b7-59ff-419c-baa1-36a74f6e7a46/volumes"
Oct 30 07:15:05.744345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.744278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:05.921263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.921228    6738 generic.go:356] "Generic (PLEG): container finished" podID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerID="9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2" exitCode=60
Oct 30 07:15:05.921442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.921319    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerDied","Data":"9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"}
Oct 30 07:15:05.921442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.921417    6738 scope.go:122] "RemoveContainer" containerID="f99234d11b53c0d0e28ce10a142fce4c20e7914bb1ba10b4912e497853a171cf"
Oct 30 07:15:05.921690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.921667    6738 scope.go:122] "RemoveContainer" containerID="9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"
Oct 30 07:15:05.922137 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.922110    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682_odh-ai-gateway-infra(bd0c2f64-d41b-4faf-86cf-e41327022c30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30"
Oct 30 07:15:05.923685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.923663    6738 generic.go:356] "Generic (PLEG): container finished" podID="b0ac0407-ad88-4786-8165-3128f9465b55" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd" exitCode=60
Oct 30 07:15:05.923788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.923751    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerDied","Data":"cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"}
Oct 30 07:15:05.924091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.924075    6738 scope.go:122] "RemoveContainer" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"
Oct 30 07:15:05.924510 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.924486    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln_odh-ai-gateway-infra(b0ac0407-ad88-4786-8165-3128f9465b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" podUID="b0ac0407-ad88-4786-8165-3128f9465b55"
Oct 30 07:15:05.926005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.925977    6738 generic.go:356] "Generic (PLEG): container finished" podID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerID="1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963" exitCode=60
Oct 30 07:15:05.926095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.926051    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerDied","Data":"1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"}
Oct 30 07:15:05.926426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.926407    6738 scope.go:122] "RemoveContainer" containerID="1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"
Oct 30 07:15:05.926774 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.926744    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m_odh-ai-gateway-infra(7c770148-a820-4ccb-a852-5f6d5bb500b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6"
Oct 30 07:15:05.928026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.928006    6738 generic.go:356] "Generic (PLEG): container finished" podID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerID="118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d" exitCode=60
Oct 30 07:15:05.928147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.928084    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerDied","Data":"118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"}
Oct 30 07:15:05.928376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.928350    6738 scope.go:122] "RemoveContainer" containerID="118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"
Oct 30 07:15:05.928682 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.928664    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf_odh-ai-gateway-infra(f1b166cb-9a21-4f88-8e50-ad75cf604db5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5"
Oct 30 07:15:05.930146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.930128    6738 scope.go:122] "RemoveContainer" containerID="4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"
Oct 30 07:15:05.930461 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.930429    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt_odh-ai-gateway-infra(e5b10586-f302-4ef0-9b02-36d388d0c6e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5"
Oct 30 07:15:05.932264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.932240    6738 generic.go:356] "Generic (PLEG): container finished" podID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerID="f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3" exitCode=60
Oct 30 07:15:05.932411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.932346    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerDied","Data":"f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"}
Oct 30 07:15:05.932583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.932568    6738 scope.go:122] "RemoveContainer" containerID="f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"
Oct 30 07:15:05.932911 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.932887    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg_odh-ai-gateway-infra(ce38fb1d-6fd8-46b4-911e-046c062e1a52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52"
Oct 30 07:15:05.934527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.934504    6738 generic.go:356] "Generic (PLEG): container finished" podID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerID="af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106" exitCode=60
Oct 30 07:15:05.934618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.934589    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerDied","Data":"af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"}
Oct 30 07:15:05.934782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.934769    6738 scope.go:122] "RemoveContainer" containerID="af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"
Oct 30 07:15:05.935030 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:05.935013    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hzf85_odh-ai-gateway-infra(a289e534-9b3f-4b4d-ad4c-7fc9dd163688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688"
Oct 30 07:15:05.939114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.939098    6738 scope.go:122] "RemoveContainer" containerID="b552b37eb831a3979f573ab9912b3d4580438d2fb9441ba7db5d8c7b9a92fc68"
Oct 30 07:15:05.954109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.954084    6738 scope.go:122] "RemoveContainer" containerID="d18d905ae057c1c43dac1e09b34ada8a7af29670ba6438b4694596241b6dc5d0"
Oct 30 07:15:05.970870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.970846    6738 scope.go:122] "RemoveContainer" containerID="9731de5e58c2aff3bbd4a922f26c8e7eafefb0c0d989ca226450ee62794339e7"
Oct 30 07:15:05.986360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:05.986332    6738 scope.go:122] "RemoveContainer" containerID="4de0734818158941b37a1cd84df64a34efa68fa641602e1e98dfe2c47f1dc4c0"
Oct 30 07:15:06.001811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.001690    6738 scope.go:122] "RemoveContainer" containerID="78ffd02833add3e7ddcf6fca3651acb83530b901587ab45ac9d8edf9fbddca50"
Oct 30 07:15:06.963151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.963121    6738 scope.go:122] "RemoveContainer" containerID="1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"
Oct 30 07:15:06.963626 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.963539    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m_odh-ai-gateway-infra(7c770148-a820-4ccb-a852-5f6d5bb500b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6"
Oct 30 07:15:06.965041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.965020    6738 scope.go:122] "RemoveContainer" containerID="118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"
Oct 30 07:15:06.965443 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.965421    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf_odh-ai-gateway-infra(f1b166cb-9a21-4f88-8e50-ad75cf604db5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5"
Oct 30 07:15:06.967003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.966984    6738 scope.go:122] "RemoveContainer" containerID="f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"
Oct 30 07:15:06.967214 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.967192    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg_odh-ai-gateway-infra(ce38fb1d-6fd8-46b4-911e-046c062e1a52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52"
Oct 30 07:15:06.968754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.968734    6738 scope.go:122] "RemoveContainer" containerID="af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"
Oct 30 07:15:06.968995 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.968973    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hzf85_odh-ai-gateway-infra(a289e534-9b3f-4b4d-ad4c-7fc9dd163688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688"
Oct 30 07:15:06.970495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.970475    6738 scope.go:122] "RemoveContainer" containerID="9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"
Oct 30 07:15:06.970702 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.970684    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682_odh-ai-gateway-infra(bd0c2f64-d41b-4faf-86cf-e41327022c30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30"
Oct 30 07:15:06.972078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:06.972062    6738 scope.go:122] "RemoveContainer" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"
Oct 30 07:15:06.972341 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:06.972281    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln_odh-ai-gateway-infra(b0ac0407-ad88-4786-8165-3128f9465b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" podUID="b0ac0407-ad88-4786-8165-3128f9465b55"
Oct 30 07:15:07.520633 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:07.520585    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:15:07.624848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:07.624810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:08.594272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:08.594229    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" podUID="dd522eb1-eda2-4129-8929-b599abf2712f" containerName="payload-processing" containerID="cri-o://2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac" gracePeriod=25
Oct 30 07:15:08.629437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:08.629377    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" podUID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" containerName="payload-pre-processing" containerID="cri-o://dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8" gracePeriod=25
Oct 30 07:15:08.939678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:08.939639    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:15:08.981883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:08.981824    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:15:09.002895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:08.999870    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\") pod \"dd522eb1-eda2-4129-8929-b599abf2712f\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") "
Oct 30 07:15:09.002895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.000124    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\" (UniqueName: \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\") pod \"dd522eb1-eda2-4129-8929-b599abf2712f\" (UID: \"dd522eb1-eda2-4129-8929-b599abf2712f\") "
Oct 30 07:15:09.002895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.000195    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\") pod \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") "
Oct 30 07:15:09.002895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.000332    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\" (UniqueName: \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\") pod \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\" (UID: \"8d14e0a8-699e-48a2-879b-b3a9ab9e771a\") "
Oct 30 07:15:09.003324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.003057    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume" pod "dd522eb1-eda2-4129-8929-b599abf2712f" (UID: "dd522eb1-eda2-4129-8929-b599abf2712f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:09.009750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.008765    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume" pod "8d14e0a8-699e-48a2-879b-b3a9ab9e771a" (UID: "8d14e0a8-699e-48a2-879b-b3a9ab9e771a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:09.012813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.012767    6738 generic.go:356] "Generic (PLEG): container finished" podID="dd522eb1-eda2-4129-8929-b599abf2712f" containerID="2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac" exitCode=0
Oct 30 07:15:09.013160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.013112    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" event={"ID":"dd522eb1-eda2-4129-8929-b599abf2712f","Type":"ContainerDied","Data":"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"}
Oct 30 07:15:09.013234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.013187    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr" event={"ID":"dd522eb1-eda2-4129-8929-b599abf2712f","Type":"ContainerDied","Data":"a96d59e620366d25cedf68f8f30521a6e2ad2bbb2e9eb1a1a533bda7853f0f73"}
Oct 30 07:15:09.013234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.013219    6738 scope.go:122] "RemoveContainer" containerID="2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"
Oct 30 07:15:09.013384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.013371    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"
Oct 30 07:15:09.018858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.018823    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff" pod "dd522eb1-eda2-4129-8929-b599abf2712f" (UID: "dd522eb1-eda2-4129-8929-b599abf2712f"). InnerVolumeSpecName "kube-api-access-gm4ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:09.019879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.019851    6738 generic.go:356] "Generic (PLEG): container finished" podID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" containerID="dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8" exitCode=0
Oct 30 07:15:09.020012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.019917    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"
Oct 30 07:15:09.020012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.019929    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" event={"ID":"8d14e0a8-699e-48a2-879b-b3a9ab9e771a","Type":"ContainerDied","Data":"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"}
Oct 30 07:15:09.020012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.019991    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q" event={"ID":"8d14e0a8-699e-48a2-879b-b3a9ab9e771a","Type":"ContainerDied","Data":"4963297fb25ef9e0d5e13d9e50dc24da986f0466a0e0445ea530f02e5ff2faee"}
Oct 30 07:15:09.025777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.025738    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56" pod "8d14e0a8-699e-48a2-879b-b3a9ab9e771a" (UID: "8d14e0a8-699e-48a2-879b-b3a9ab9e771a"). InnerVolumeSpecName "kube-api-access-l9c56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:09.037170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.037037    6738 scope.go:122] "RemoveContainer" containerID="2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"
Oct 30 07:15:09.037603 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:09.037565    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac\": container with ID starting with 2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac not found: ID does not exist" containerID="2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"
Oct 30 07:15:09.037731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.037615    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac"} err="failed to get container status \"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac\": rpc error: code = NotFound desc = could not find container \"2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac\": container with ID starting with 2f93b3ae3f7f7936799cc7d11119cb4ae50d82891fe1b04da814c74285dca4ac not found: ID does not exist"
Oct 30 07:15:09.037731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.037644    6738 scope.go:122] "RemoveContainer" containerID="dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"
Oct 30 07:15:09.053541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.053416    6738 scope.go:122] "RemoveContainer" containerID="dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"
Oct 30 07:15:09.053856 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:09.053829    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8\": container with ID starting with dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8 not found: ID does not exist" containerID="dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"
Oct 30 07:15:09.053934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.053870    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8"} err="failed to get container status \"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8\": rpc error: code = NotFound desc = could not find container \"dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8\": container with ID starting with dc5cf744ff355f0e042c34768c209cac9b7a2e762c61150ce95f958f8312eae8 not found: ID does not exist"
Oct 30 07:15:09.103016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.102976    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9c56\" (UniqueName: \"kubernetes.io/projected/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-kube-api-access-l9c56\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:09.103016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.103009    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd522eb1-eda2-4129-8929-b599abf2712f-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:09.103016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.103022    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm4ff\" (UniqueName: \"kubernetes.io/projected/dd522eb1-eda2-4129-8929-b599abf2712f-kube-api-access-gm4ff\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:09.103264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.103035    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d14e0a8-699e-48a2-879b-b3a9ab9e771a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:09.265439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.265346    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:09.363244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.363198    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:15:09.367807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.367783    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c7884-56db767db-h7gsr"]
Oct 30 07:15:09.401784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.401744    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:15:09.406258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:09.406231    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c7884-84d669485f-8m94q"]
Oct 30 07:15:10.029807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.029757    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" path="/var/lib/kubelet/pods/8d14e0a8-699e-48a2-879b-b3a9ab9e771a/volumes"
Oct 30 07:15:10.031603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.031562    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd522eb1-eda2-4129-8929-b599abf2712f" path="/var/lib/kubelet/pods/dd522eb1-eda2-4129-8929-b599abf2712f/volumes"
Oct 30 07:15:10.051589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.051539    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:10.194429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.194389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:10.239427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.239387    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"]
Oct 30 07:15:10.266021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.265974    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3bda71-ffjzc"]
Oct 30 07:15:10.316372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.314518    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:10.802244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:10.802212    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:11.691314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:11.691274    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b9c30b-e4e8-42b2-8d9f-ed32799c0346" path="/var/lib/kubelet/pods/e8b9c30b-e4e8-42b2-8d9f-ed32799c0346/volumes"
Oct 30 07:15:12.246164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:12.246126    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:14.095658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:14.095614    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:14.683313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:14.683271    6738 scope.go:122] "RemoveContainer" containerID="4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"
Oct 30 07:15:15.095907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:15.095869    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerStarted","Data":"87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b"}
Oct 30 07:15:15.411019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:15.410972    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:15.697061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:15.694122    6738 scope.go:122] "RemoveContainer" containerID="118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"
Oct 30 07:15:15.698774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:15.698268    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:15.778199 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:15.776467    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b10586_f302_4ef0_9b02_36d388d0c6e5.slice/crio-conmon-87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 07:15:16.142829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.142266    6738 generic.go:356] "Generic (PLEG): container finished" podID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerID="87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b" exitCode=60
Oct 30 07:15:16.142829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.142366    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerDied","Data":"87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b"}
Oct 30 07:15:16.142829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.142423    6738 scope.go:122] "RemoveContainer" containerID="4eff313bf5aaee9fc0ea41da9fe580af1cfb1f2e135d104f2d81b1de7df77aa8"
Oct 30 07:15:16.143314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.143016    6738 scope.go:122] "RemoveContainer" containerID="87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b"
Oct 30 07:15:16.147684 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:16.143885    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt_odh-ai-gateway-infra(e5b10586-f302-4ef0-9b02-36d388d0c6e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5"
Oct 30 07:15:16.149348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.148125    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:15:16.151417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151361    6738 cpu_manager.go:414] "containerMap: removing container" podUID="dd522eb1-eda2-4129-8929-b599abf2712f" containerName="payload-processing"
Oct 30 07:15:16.151417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151391    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd522eb1-eda2-4129-8929-b599abf2712f" containerName="payload-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151449    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" containerName="payload-pre-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151457    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" containerName="payload-pre-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151489    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" containerName="payload-pre-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151497    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" containerName="payload-pre-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151515    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" containerName="payload-processing"
Oct 30 07:15:16.151639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151522    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" containerName="payload-processing"
Oct 30 07:15:16.153592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151718    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d14e0a8-699e-48a2-879b-b3a9ab9e771a" containerName="payload-pre-processing"
Oct 30 07:15:16.153592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151731    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0552d7cc-b654-4de8-af6f-fd26ac21f86d" containerName="payload-processing"
Oct 30 07:15:16.153592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151752    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd522eb1-eda2-4129-8929-b599abf2712f" containerName="payload-processing"
Oct 30 07:15:16.153592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.151766    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5050b7-59ff-419c-baa1-36a74f6e7a46" containerName="payload-pre-processing"
Oct 30 07:15:16.158979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.157590    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:16.158979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.157713    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.164864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.163410    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:15:16.176591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.176558    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a273fd-dockercfg-fwrn5\""
Oct 30 07:15:16.177050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.176997    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a273fd\""
Oct 30 07:15:16.316127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.316022    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dts\" (UniqueName: \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.316349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.316231    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.417347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.417304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.417576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.417560    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dts\" (UniqueName: \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.418049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.418031    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.440935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.440902    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dts\" (UniqueName: \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\") pod \"payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.485091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.485046    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:16.689002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.688971    6738 scope.go:122] "RemoveContainer" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"
Oct 30 07:15:16.762397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.762364    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:16.791693 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:16.791656    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09954160_f955_4bd7_b032_6f723e09d687.slice/crio-7f9dab65eb5c254d6ef4990be0d9054f8aa3357aa831f6b2598add8bfcc1b703 WatchSource:0}: Error finding container 7f9dab65eb5c254d6ef4990be0d9054f8aa3357aa831f6b2598add8bfcc1b703: Status 404 returned error can't find the container with id 7f9dab65eb5c254d6ef4990be0d9054f8aa3357aa831f6b2598add8bfcc1b703
Oct 30 07:15:16.793281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.793256    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:15:16.793820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.793803    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:15:16.876711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:16.876671    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:17.002370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.000846    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:17.155660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.155630    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" event={"ID":"09954160-f955-4bd7-b032-6f723e09d687","Type":"ContainerStarted","Data":"7f9dab65eb5c254d6ef4990be0d9054f8aa3357aa831f6b2598add8bfcc1b703"}
Oct 30 07:15:17.157919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.157887    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerStarted","Data":"38b8c65e52e7ed56da3f3140972176db72644f36d45c003007a5ab494b3a929c"}
Oct 30 07:15:17.159955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.159933    6738 generic.go:356] "Generic (PLEG): container finished" podID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerID="4b642f9418db77e75c85a86d01e4369600d9e5504639f293272ca6b0351bccf6" exitCode=60
Oct 30 07:15:17.160066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.159982    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerDied","Data":"4b642f9418db77e75c85a86d01e4369600d9e5504639f293272ca6b0351bccf6"}
Oct 30 07:15:17.160066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.160019    6738 scope.go:122] "RemoveContainer" containerID="118d4b79a9d8aab0f1ba410fd139e5d947bd8fa5d561ba7e0c89f038df79e35d"
Oct 30 07:15:17.160400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.160385    6738 scope.go:122] "RemoveContainer" containerID="4b642f9418db77e75c85a86d01e4369600d9e5504639f293272ca6b0351bccf6"
Oct 30 07:15:17.160759 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:17.160726    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf_odh-ai-gateway-infra(f1b166cb-9a21-4f88-8e50-ad75cf604db5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5"
Oct 30 07:15:17.336436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.336405    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:17.441410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.441370    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\") pod \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") "
Oct 30 07:15:17.441603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.441586    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\" (UniqueName: \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\") pod \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\" (UID: \"e5b10586-f302-4ef0-9b02-36d388d0c6e5\") "
Oct 30 07:15:17.442144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.441784    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca" pod "e5b10586-f302-4ef0-9b02-36d388d0c6e5" (UID: "e5b10586-f302-4ef0-9b02-36d388d0c6e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:17.442308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.442268    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b10586-f302-4ef0-9b02-36d388d0c6e5-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:17.444513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.444485    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8" pod "e5b10586-f302-4ef0-9b02-36d388d0c6e5" (UID: "e5b10586-f302-4ef0-9b02-36d388d0c6e5"). InnerVolumeSpecName "kube-api-access-78vw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:17.544550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.544514    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vw8\" (UniqueName: \"kubernetes.io/projected/e5b10586-f302-4ef0-9b02-36d388d0c6e5-kube-api-access-78vw8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:17.624984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.624948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:17.682853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.682819    6738 scope.go:122] "RemoveContainer" containerID="1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"
Oct 30 07:15:17.683053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.682973    6738 scope.go:122] "RemoveContainer" containerID="f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"
Oct 30 07:15:17.790690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.790655    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:17.967139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.967100    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:15:17.967460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.967423    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" containerID="cri-o://b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220" gracePeriod=30
Oct 30 07:15:17.999057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:17.999011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:18.233399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.233035    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt" event={"ID":"e5b10586-f302-4ef0-9b02-36d388d0c6e5","Type":"ContainerDied","Data":"b3035b38293851732cebee027bac937542f1b17dacdfd0d8a89809e775feaec6"}
Oct 30 07:15:18.233399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.233212    6738 scope.go:122] "RemoveContainer" containerID="87114d442a3a62b02faa401f822042f55257e270143bc26fae03cf2b629bab1b"
Oct 30 07:15:18.233879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.233423    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"
Oct 30 07:15:18.306463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.306425    6738 generic.go:356] "Generic (PLEG): container finished" podID="b0ac0407-ad88-4786-8165-3128f9465b55" containerID="38b8c65e52e7ed56da3f3140972176db72644f36d45c003007a5ab494b3a929c" exitCode=60
Oct 30 07:15:18.306716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.306689    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerDied","Data":"38b8c65e52e7ed56da3f3140972176db72644f36d45c003007a5ab494b3a929c"}
Oct 30 07:15:18.307721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.307414    6738 scope.go:122] "RemoveContainer" containerID="38b8c65e52e7ed56da3f3140972176db72644f36d45c003007a5ab494b3a929c"
Oct 30 07:15:18.308379 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:18.308344    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln_odh-ai-gateway-infra(b0ac0407-ad88-4786-8165-3128f9465b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" podUID="b0ac0407-ad88-4786-8165-3128f9465b55"
Oct 30 07:15:18.344629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.344590    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:18.374316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.369739    6738 scope.go:122] "RemoveContainer" containerID="cd39f7bf25abc8a7018d1058c15844fe6d50b45aed9b9722785bfadd12c87acd"
Oct 30 07:15:18.381344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.380360    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:18.391549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.390992    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:18.405360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.400849    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c0e3de-29889075-b86lt"]
Oct 30 07:15:18.616258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.615530    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"]
Oct 30 07:15:18.616699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.616674    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.616768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.616703    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.616768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.616744    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.616768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.616753    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.617018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.617003    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.617072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.617026    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.617072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.617044    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:18.643401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.643372    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"]
Oct 30 07:15:18.643612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.643521    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.679063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.679035    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:18.683797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.683771    6738 scope.go:122] "RemoveContainer" containerID="af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"
Oct 30 07:15:18.707277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.707244    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.707805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.707774    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.707920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.707818    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvlm\" (UniqueName: \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.760566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.760485    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:15:18.809822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.809755    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\") pod \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") "
Oct 30 07:15:18.809822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.809824    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\" (UniqueName: \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\") pod \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\" (UID: \"f1b166cb-9a21-4f88-8e50-ad75cf604db5\") "
Oct 30 07:15:18.810191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.810166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.810643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.810570    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.810643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.810614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvlm\" (UniqueName: \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.812212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.811589    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca" pod "f1b166cb-9a21-4f88-8e50-ad75cf604db5" (UID: "f1b166cb-9a21-4f88-8e50-ad75cf604db5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:18.812713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.812687    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.875345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.868855    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvlm\" (UniqueName: \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.875345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.873846    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f357db-tn2rn\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:18.878510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.878346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz" pod "f1b166cb-9a21-4f88-8e50-ad75cf604db5" (UID: "f1b166cb-9a21-4f88-8e50-ad75cf604db5"). InnerVolumeSpecName "kube-api-access-7chwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:18.938314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.935537    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b166cb-9a21-4f88-8e50-ad75cf604db5-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:18.938314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.935709    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7chwz\" (UniqueName: \"kubernetes.io/projected/f1b166cb-9a21-4f88-8e50-ad75cf604db5-kube-api-access-7chwz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:18.997030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:18.996719    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:19.293191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.293097    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:19.329446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.324977    6738 generic.go:356] "Generic (PLEG): container finished" podID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerID="f6199ce060ebcbcd44d0babb3cc7383c789206dd09a1ec2ec49a2c1c020c80c1" exitCode=60
Oct 30 07:15:19.329446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.325097    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerDied","Data":"f6199ce060ebcbcd44d0babb3cc7383c789206dd09a1ec2ec49a2c1c020c80c1"}
Oct 30 07:15:19.329446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.325156    6738 scope.go:122] "RemoveContainer" containerID="f3ef3265a7c6eefa270c744299e3e7c869676988c7132e79c8ee43488d7d06a3"
Oct 30 07:15:19.329446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.325714    6738 scope.go:122] "RemoveContainer" containerID="f6199ce060ebcbcd44d0babb3cc7383c789206dd09a1ec2ec49a2c1c020c80c1"
Oct 30 07:15:19.329446 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:19.326165    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg_odh-ai-gateway-infra(ce38fb1d-6fd8-46b4-911e-046c062e1a52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52"
Oct 30 07:15:19.358644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.332341    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerStarted","Data":"cfd726ef0900d4100c17dc13a7d89417c1104cd08855fdee6e2eba0e6ac33b7c"}
Oct 30 07:15:19.420478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.406005    6738 generic.go:356] "Generic (PLEG): container finished" podID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerID="9c7b4193c1043c3f00a3f0b653ed01453113cde5e303ec36ecefc961c66ba0aa" exitCode=60
Oct 30 07:15:19.420478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.406113    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerDied","Data":"9c7b4193c1043c3f00a3f0b653ed01453113cde5e303ec36ecefc961c66ba0aa"}
Oct 30 07:15:19.420478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.406689    6738 scope.go:122] "RemoveContainer" containerID="9c7b4193c1043c3f00a3f0b653ed01453113cde5e303ec36ecefc961c66ba0aa"
Oct 30 07:15:19.420478 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:19.407124    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m_odh-ai-gateway-infra(7c770148-a820-4ccb-a852-5f6d5bb500b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6"
Oct 30 07:15:19.454184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.452863    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf" event={"ID":"f1b166cb-9a21-4f88-8e50-ad75cf604db5","Type":"ContainerDied","Data":"df90cbcf52af070111876cc2bbe8492be3895f60aacec7e36a28c1d7c345f764"}
Oct 30 07:15:19.454184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.453004    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"
Oct 30 07:15:19.467180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.467100    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"]
Oct 30 07:15:19.469947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.469840    6738 scope.go:122] "RemoveContainer" containerID="1b41bb5357bcceb2aa72656762a5e6b1d7241c05bf88dad3eb0b92d742a78963"
Oct 30 07:15:19.500461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.500434    6738 scope.go:122] "RemoveContainer" containerID="4b642f9418db77e75c85a86d01e4369600d9e5504639f293272ca6b0351bccf6"
Oct 30 07:15:19.588206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.588159    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:19.593388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.593357    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-06b26e-29889075-2l4kf"]
Oct 30 07:15:19.689466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.689431    6738 scope.go:122] "RemoveContainer" containerID="9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"
Oct 30 07:15:19.695051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.695021    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" path="/var/lib/kubelet/pods/e5b10586-f302-4ef0-9b02-36d388d0c6e5/volumes"
Oct 30 07:15:19.695659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.695635    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" path="/var/lib/kubelet/pods/f1b166cb-9a21-4f88-8e50-ad75cf604db5/volumes"
Oct 30 07:15:19.698894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.698874    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:19.784207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.784169    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\") pod \"b0ac0407-ad88-4786-8165-3128f9465b55\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") "
Oct 30 07:15:19.784366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.784233    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\" (UniqueName: \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\") pod \"b0ac0407-ad88-4786-8165-3128f9465b55\" (UID: \"b0ac0407-ad88-4786-8165-3128f9465b55\") "
Oct 30 07:15:19.784555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.784526    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca" pod "b0ac0407-ad88-4786-8165-3128f9465b55" (UID: "b0ac0407-ad88-4786-8165-3128f9465b55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:19.785092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.785068    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac0407-ad88-4786-8165-3128f9465b55-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:19.787953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.787921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t" pod "b0ac0407-ad88-4786-8165-3128f9465b55" (UID: "b0ac0407-ad88-4786-8165-3128f9465b55"). InnerVolumeSpecName "kube-api-access-llg7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:19.886386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:19.886358    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llg7t\" (UniqueName: \"kubernetes.io/projected/b0ac0407-ad88-4786-8165-3128f9465b55-kube-api-access-llg7t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:20.211986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.211953    6738 scope.go:122] "RemoveContainer" containerID="b03ff548d872faf864a0ded649c207e2faaca3ea1cb8edf5889f9e55f52f6422"
Oct 30 07:15:20.223461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.223438    6738 scope.go:122] "RemoveContainer" containerID="a42a8f7d5391230ef5a22e732c54355955bf884b2e5c2e7ba65ed6505b68c84e"
Oct 30 07:15:20.234073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.234048    6738 scope.go:122] "RemoveContainer" containerID="28bb002af912e63da1d27f28b8c6036947f8f9702e1490087c2c9e3e2b9a33d2"
Oct 30 07:15:20.245699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.245673    6738 scope.go:122] "RemoveContainer" containerID="56579fea62f960c19ec50270db245ff2783ccfec8e612de69ffdad7b50cd7c62"
Oct 30 07:15:20.256520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.256499    6738 scope.go:122] "RemoveContainer" containerID="5a7558a06b1d954a810ed0e06876edf3c61c152423dc64916eac5c9f1457534a"
Oct 30 07:15:20.269909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.269878    6738 scope.go:122] "RemoveContainer" containerID="54b79d8e5e79fa6483e87052841937834e8567e16b8fbab70f2dcb6ba6317525"
Oct 30 07:15:20.283262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.283232    6738 scope.go:122] "RemoveContainer" containerID="d1877e2b8db6937c7084e2d4d5a737811db03a8cd2c10a6e529a1e7a06337c19"
Oct 30 07:15:20.293725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.293706    6738 scope.go:122] "RemoveContainer" containerID="ac9fc93e1cba162ab7dd5979c37a3b07a7edbdd6ab001e99a366bff96b673b8a"
Oct 30 07:15:20.397435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.397403    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:20.413432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.413234    6738 patch_prober.go:29] interesting pod/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:15:20.413432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.413286    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:15:20.487003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.486965    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" event={"ID":"09954160-f955-4bd7-b032-6f723e09d687","Type":"ContainerStarted","Data":"7443399f1d72e973672bd215ace78ede3e6a93b7104dac5e84505822e1c96c63"}
Oct 30 07:15:20.487166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.487068    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:20.493428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.493398    6738 generic.go:356] "Generic (PLEG): container finished" podID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerID="cfd726ef0900d4100c17dc13a7d89417c1104cd08855fdee6e2eba0e6ac33b7c" exitCode=60
Oct 30 07:15:20.493554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.493434    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerDied","Data":"cfd726ef0900d4100c17dc13a7d89417c1104cd08855fdee6e2eba0e6ac33b7c"}
Oct 30 07:15:20.493554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.493509    6738 scope.go:122] "RemoveContainer" containerID="af07e0651d26b7261a76e8f0c72f2015e9a5a7419191bb9716192df546795106"
Oct 30 07:15:20.493984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.493961    6738 scope.go:122] "RemoveContainer" containerID="cfd726ef0900d4100c17dc13a7d89417c1104cd08855fdee6e2eba0e6ac33b7c"
Oct 30 07:15:20.495062 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:20.494382    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hzf85_odh-ai-gateway-infra(a289e534-9b3f-4b4d-ad4c-7fc9dd163688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688"
Oct 30 07:15:20.496481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.496461    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:20.497196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.497175    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerStarted","Data":"cb28cff00c1d0646911ef15d8006a7cef516a8fe3d5b82912977fa0fd6bd57b0"}
Oct 30 07:15:20.499007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.498936    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"
Oct 30 07:15:20.499007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.498993    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln" event={"ID":"b0ac0407-ad88-4786-8165-3128f9465b55","Type":"ContainerDied","Data":"9640aa3dcf048e49289777b4f522c79b5f737c7cb46962f8c7e1d52aa8349782"}
Oct 30 07:15:20.504966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.504940    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn" event={"ID":"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19","Type":"ContainerStarted","Data":"ee813825d5dd0df799c8349e8c0e64f0b3ec90a2ebcf48ad782205b611ba26a4"}
Oct 30 07:15:20.505077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.504983    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn" event={"ID":"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19","Type":"ContainerStarted","Data":"52c31018a8c7826ef1c29bbe8d7344077fc828b1b0ba4859d119457fef856dfe"}
Oct 30 07:15:20.543387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.543275    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" podStartSLOduration=1.5298277260000002 podStartE2EDuration="4.54325433s" totalImagesPullingTime="3.013426604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:20.534570197 +0000 UTC m=+1080.967334091" watchObservedRunningTime="2026-10-30 07:15:20.54325433 +0000 UTC m=+1080.976018224"
Oct 30 07:15:20.545652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.544731    6738 scope.go:122] "RemoveContainer" containerID="38b8c65e52e7ed56da3f3140972176db72644f36d45c003007a5ab494b3a929c"
Oct 30 07:15:20.547580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.547557    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:20.860369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.860339    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:20.868490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.868431    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a273fd-29889075-zg6ln"]
Oct 30 07:15:20.900078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.899997    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:20.902194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:20.902164    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.011859    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\") pod \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") "
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.012012    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\" (UniqueName: \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\") pod \"7c770148-a820-4ccb-a852-5f6d5bb500b6\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") "
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.012196    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\" (UniqueName: \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\") pod \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\" (UID: \"ce38fb1d-6fd8-46b4-911e-046c062e1a52\") "
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.012240    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\") pod \"7c770148-a820-4ccb-a852-5f6d5bb500b6\" (UID: \"7c770148-a820-4ccb-a852-5f6d5bb500b6\") "
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.013172    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca" pod "7c770148-a820-4ccb-a852-5f6d5bb500b6" (UID: "7c770148-a820-4ccb-a852-5f6d5bb500b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:21.013952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.013543    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca" pod "ce38fb1d-6fd8-46b4-911e-046c062e1a52" (UID: "ce38fb1d-6fd8-46b4-911e-046c062e1a52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:21.016133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.016098    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r" pod "ce38fb1d-6fd8-46b4-911e-046c062e1a52" (UID: "ce38fb1d-6fd8-46b4-911e-046c062e1a52"). InnerVolumeSpecName "kube-api-access-2946r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:21.018441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.018410    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq" pod "7c770148-a820-4ccb-a852-5f6d5bb500b6" (UID: "7c770148-a820-4ccb-a852-5f6d5bb500b6"). InnerVolumeSpecName "kube-api-access-79dxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:21.114539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.114504    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dxq\" (UniqueName: \"kubernetes.io/projected/7c770148-a820-4ccb-a852-5f6d5bb500b6-kube-api-access-79dxq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.114539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.114535    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2946r\" (UniqueName: \"kubernetes.io/projected/ce38fb1d-6fd8-46b4-911e-046c062e1a52-kube-api-access-2946r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.114539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.114545    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c770148-a820-4ccb-a852-5f6d5bb500b6-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.114539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.114554    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce38fb1d-6fd8-46b4-911e-046c062e1a52-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.515798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.515767    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"
Oct 30 07:15:21.516363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.515789    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m" event={"ID":"7c770148-a820-4ccb-a852-5f6d5bb500b6","Type":"ContainerDied","Data":"a941dc9d97c0ba215a26b900338e98bc88cd96dd96f879f497d513685672ec0e"}
Oct 30 07:15:21.516486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.516419    6738 scope.go:122] "RemoveContainer" containerID="9c7b4193c1043c3f00a3f0b653ed01453113cde5e303ec36ecefc961c66ba0aa"
Oct 30 07:15:21.517775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.517737    6738 generic.go:356] "Generic (PLEG): container finished" podID="b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" containerID="ee813825d5dd0df799c8349e8c0e64f0b3ec90a2ebcf48ad782205b611ba26a4" exitCode=0
Oct 30 07:15:21.517881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.517853    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn" event={"ID":"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19","Type":"ContainerDied","Data":"ee813825d5dd0df799c8349e8c0e64f0b3ec90a2ebcf48ad782205b611ba26a4"}
Oct 30 07:15:21.521316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.520024    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"
Oct 30 07:15:21.521316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.520030    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg" event={"ID":"ce38fb1d-6fd8-46b4-911e-046c062e1a52","Type":"ContainerDied","Data":"36fa7144021062611dd613aed9a76118822d8aa81b7ebed72fe84245abd020c7"}
Oct 30 07:15:21.532256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.532228    6738 scope.go:122] "RemoveContainer" containerID="f6199ce060ebcbcd44d0babb3cc7383c789206dd09a1ec2ec49a2c1c020c80c1"
Oct 30 07:15:21.532441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.532416    6738 generic.go:356] "Generic (PLEG): container finished" podID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerID="cb28cff00c1d0646911ef15d8006a7cef516a8fe3d5b82912977fa0fd6bd57b0" exitCode=60
Oct 30 07:15:21.532519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.532443    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerDied","Data":"cb28cff00c1d0646911ef15d8006a7cef516a8fe3d5b82912977fa0fd6bd57b0"}
Oct 30 07:15:21.534207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.532965    6738 scope.go:122] "RemoveContainer" containerID="cb28cff00c1d0646911ef15d8006a7cef516a8fe3d5b82912977fa0fd6bd57b0"
Oct 30 07:15:21.534207 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:21.533449    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682_odh-ai-gateway-infra(bd0c2f64-d41b-4faf-86cf-e41327022c30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30"
Oct 30 07:15:21.547129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.547019    6738 scope.go:122] "RemoveContainer" containerID="9a1642ba0fe05ba730c26f3b654a21151f577b8d121061cac7e096e058c41ac2"
Oct 30 07:15:21.600135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.599408    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:21.635756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.635639    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:21.648914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.648875    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7c7884-29889075-6lmwg"]
Oct 30 07:15:21.691247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.691220    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:21.705471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.705258    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" path="/var/lib/kubelet/pods/b0ac0407-ad88-4786-8165-3128f9465b55/volumes"
Oct 30 07:15:21.706428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.706404    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" path="/var/lib/kubelet/pods/ce38fb1d-6fd8-46b4-911e-046c062e1a52/volumes"
Oct 30 07:15:21.732581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.732547    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:21.741494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.741394    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889075-k8m6m"]
Oct 30 07:15:21.760509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.760477    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:21.811575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.811540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:21.857331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.857264    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\" (UniqueName: \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\") pod \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") "
Oct 30 07:15:21.857331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.857337    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\") pod \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") "
Oct 30 07:15:21.857562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.857488    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\") pod \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\" (UID: \"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19\") "
Oct 30 07:15:21.880031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.879641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm" pod "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" (UID: "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19"). InnerVolumeSpecName "kube-api-access-frvlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:21.892922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.892092    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca" pod "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" (UID: "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:15:21.896680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.895535    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca" pod "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" (UID: "b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:21.975473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.975429    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\" (UniqueName: \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\") pod \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") "
Oct 30 07:15:21.975681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.975608    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\") pod \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\" (UID: \"a289e534-9b3f-4b4d-ad4c-7fc9dd163688\") "
Oct 30 07:15:21.975987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.975961    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca" pod "a289e534-9b3f-4b4d-ad4c-7fc9dd163688" (UID: "a289e534-9b3f-4b4d-ad4c-7fc9dd163688"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:21.976234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.976218    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.976343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.976235    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.976343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.976245    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvlm\" (UniqueName: \"kubernetes.io/projected/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-kube-api-access-frvlm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.976343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.976254    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:21.977693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:21.977668    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx" pod "a289e534-9b3f-4b4d-ad4c-7fc9dd163688" (UID: "a289e534-9b3f-4b4d-ad4c-7fc9dd163688"). InnerVolumeSpecName "kube-api-access-46pmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:22.077560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.077475    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pmx\" (UniqueName: \"kubernetes.io/projected/a289e534-9b3f-4b4d-ad4c-7fc9dd163688-kube-api-access-46pmx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:22.548731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.548704    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"
Oct 30 07:15:22.549252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.548763    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn" event={"ID":"b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19","Type":"ContainerDied","Data":"52c31018a8c7826ef1c29bbe8d7344077fc828b1b0ba4859d119457fef856dfe"}
Oct 30 07:15:22.549252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.548820    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c31018a8c7826ef1c29bbe8d7344077fc828b1b0ba4859d119457fef856dfe"
Oct 30 07:15:22.555730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.555700    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85" event={"ID":"a289e534-9b3f-4b4d-ad4c-7fc9dd163688","Type":"ContainerDied","Data":"01d16adb6890bc726bad5361234453c1458f41b88e414f37cc22b915d5cd8187"}
Oct 30 07:15:22.555858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.555730    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"
Oct 30 07:15:22.555858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.555760    6738 scope.go:122] "RemoveContainer" containerID="cfd726ef0900d4100c17dc13a7d89417c1104cd08855fdee6e2eba0e6ac33b7c"
Oct 30 07:15:22.649841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.649751    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:22.654959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.654934    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hzf85"]
Oct 30 07:15:22.721509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.721485    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:22.840978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.840880    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\" (UniqueName: \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\") pod \"bd0c2f64-d41b-4faf-86cf-e41327022c30\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") "
Oct 30 07:15:22.840978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.840931    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\") pod \"bd0c2f64-d41b-4faf-86cf-e41327022c30\" (UID: \"bd0c2f64-d41b-4faf-86cf-e41327022c30\") "
Oct 30 07:15:22.841425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.841402    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca" pod "bd0c2f64-d41b-4faf-86cf-e41327022c30" (UID: "bd0c2f64-d41b-4faf-86cf-e41327022c30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:22.841540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.841529    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0c2f64-d41b-4faf-86cf-e41327022c30-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:22.843099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.843073    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52" pod "bd0c2f64-d41b-4faf-86cf-e41327022c30" (UID: "bd0c2f64-d41b-4faf-86cf-e41327022c30"). InnerVolumeSpecName "kube-api-access-7km52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:22.942528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:22.942487    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7km52\" (UniqueName: \"kubernetes.io/projected/bd0c2f64-d41b-4faf-86cf-e41327022c30-kube-api-access-7km52\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.257273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.257238    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:15:23.348126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348080    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348149    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348195    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348257    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348348    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348405    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348458    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348495    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348561    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\") pod \"90706db2-1f6c-4280-8277-14a0a3a82afb\" (UID: \"90706db2-1f6c-4280-8277-14a0a3a82afb\") "
Oct 30 07:15:23.348963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348929    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:23.349044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.348945    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:23.349223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349200    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:23.349434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:23.349589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349565    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.349589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349593    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.349739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349608    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.349739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349622    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90706db2-1f6c-4280-8277-14a0a3a82afb-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.349849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.349772    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:23.351717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.351669    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:23.351836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.351793    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:23.351955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.351938    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:15:23.352170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.352143    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5" pod "90706db2-1f6c-4280-8277-14a0a3a82afb" (UID: "90706db2-1f6c-4280-8277-14a0a3a82afb"). InnerVolumeSpecName "kube-api-access-rdzc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:23.451239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.451151    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.451239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.451188    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.451239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.451198    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.451239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.451207    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzc5\" (UniqueName: \"kubernetes.io/projected/90706db2-1f6c-4280-8277-14a0a3a82afb-kube-api-access-rdzc5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.451239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.451216    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90706db2-1f6c-4280-8277-14a0a3a82afb-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.565691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.565654    6738 generic.go:356] "Generic (PLEG): container finished" podID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerID="b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220" exitCode=0
Oct 30 07:15:23.566192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.565711    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"
Oct 30 07:15:23.566192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.565738    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerDied","Data":"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"}
Oct 30 07:15:23.566192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.565828    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq" event={"ID":"90706db2-1f6c-4280-8277-14a0a3a82afb","Type":"ContainerDied","Data":"b020bfec877e95f1fb17ea952def4c8aaac35ee83fb4188dcebc7b981992c724"}
Oct 30 07:15:23.566192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.565857    6738 scope.go:122] "RemoveContainer" containerID="b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"
Oct 30 07:15:23.568887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.568860    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682" event={"ID":"bd0c2f64-d41b-4faf-86cf-e41327022c30","Type":"ContainerDied","Data":"3c3d8b64ff408305dce3a99ad1ac616a4837a0697462f8b89e741b7d8a456c8b"}
Oct 30 07:15:23.569016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.568889    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"
Oct 30 07:15:23.579593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.579567    6738 scope.go:122] "RemoveContainer" containerID="6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"
Oct 30 07:15:23.590932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.590905    6738 scope.go:122] "RemoveContainer" containerID="b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"
Oct 30 07:15:23.591287 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:23.591264    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220\": container with ID starting with b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220 not found: ID does not exist" containerID="b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"
Oct 30 07:15:23.591386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.591312    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220"} err="failed to get container status \"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220\": rpc error: code = NotFound desc = could not find container \"b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220\": container with ID starting with b066f1ee97fbf43dabf72adf7d8dc486778506bf093456890b5661979f131220 not found: ID does not exist"
Oct 30 07:15:23.591386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.591333    6738 scope.go:122] "RemoveContainer" containerID="6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"
Oct 30 07:15:23.591619 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:23.591601    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501\": container with ID starting with 6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501 not found: ID does not exist" containerID="6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"
Oct 30 07:15:23.591678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.591629    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501"} err="failed to get container status \"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501\": rpc error: code = NotFound desc = could not find container \"6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501\": container with ID starting with 6297d74b894ab8de32e5ac4d955f0a3bfb7f7898382e67115f5c364d4c99a501 not found: ID does not exist"
Oct 30 07:15:23.591678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.591652    6738 scope.go:122] "RemoveContainer" containerID="cb28cff00c1d0646911ef15d8006a7cef516a8fe3d5b82912977fa0fd6bd57b0"
Oct 30 07:15:23.622109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.622074    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:15:23.634028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.633996    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3bda71-openshift-default-c56fd4d6f-w24bq"]
Oct 30 07:15:23.668429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.668394    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:23.672519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.672495    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889075-sl682"]
Oct 30 07:15:23.691414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.691377    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" path="/var/lib/kubelet/pods/7c770148-a820-4ccb-a852-5f6d5bb500b6/volumes"
Oct 30 07:15:23.692027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.692004    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" path="/var/lib/kubelet/pods/90706db2-1f6c-4280-8277-14a0a3a82afb/volumes"
Oct 30 07:15:23.692528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.692509    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" path="/var/lib/kubelet/pods/a289e534-9b3f-4b4d-ad4c-7fc9dd163688/volumes"
Oct 30 07:15:23.693434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.693413    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" path="/var/lib/kubelet/pods/bd0c2f64-d41b-4faf-86cf-e41327022c30/volumes"
Oct 30 07:15:23.747988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:23.747458    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:25.647840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:25.647801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:27.700793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:27.700757    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:29.096542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:29.096445    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:15:29.130006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:29.129972    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:15:29.243230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:29.243184    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:29.374393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:29.373879    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:30.110968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:30.110933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:31.537246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:31.537211    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:15:32.032236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:32.032198    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"]
Oct 30 07:15:32.036520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:32.036467    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-508552a7-bhgbt"]
Oct 30 07:15:32.226813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:32.226778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:33.385983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.385948    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7430-e8fc66"]
Oct 30 07:15:33.386663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386645    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386666    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386676    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386681    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386687    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386692    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386702    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386707    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.386718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386718    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386723    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b10586-f302-4ef0-9b02-36d388d0c6e5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386731    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386736    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386745    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386749    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386760    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386764    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386775    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386781    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386788    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386792    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386804    6738 cpu_manager.go:414] "containerMap: removing container" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386808    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386816    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386820    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386827    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386831    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386839    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" containerName="revoke-keys"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386843    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" containerName="revoke-keys"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386851    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386855    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386862    6738 cpu_manager.go:414] "containerMap: removing container" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386866    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386871    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386875    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386890    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386894    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386905    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.386960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.386909    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387024    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387033    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387038    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387044    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387050    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387058    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387064    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387069    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387075    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387081    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387086    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387091    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a289e534-9b3f-4b4d-ad4c-7fc9dd163688" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387097    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" containerName="revoke-keys"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387102    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:33.387922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.387108    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:33.392995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.392965    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:33.407128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.407097    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7430-e8fc66"]
Oct 30 07:15:33.414106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.414083    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5q8j4\""
Oct 30 07:15:33.421551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.421514    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p79hj\" (UniqueName: \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\") pod \"test-curl-7430-e8fc66\" (UID: \"49f1933d-5f12-494f-bb6d-80879b1899b2\") " pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:33.522728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.522691    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p79hj\" (UniqueName: \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\") pod \"test-curl-7430-e8fc66\" (UID: \"49f1933d-5f12-494f-bb6d-80879b1899b2\") " pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:33.546279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.546242    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p79hj\" (UniqueName: \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\") pod \"test-curl-7430-e8fc66\" (UID: \"49f1933d-5f12-494f-bb6d-80879b1899b2\") " pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:33.691972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.691876    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebcde54c-2656-4870-abda-67b1d81d15d9" path="/var/lib/kubelet/pods/ebcde54c-2656-4870-abda-67b1d81d15d9/volumes"
Oct 30 07:15:33.708977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:33.708938    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:34.022691 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:34.022659    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f1933d_5f12_494f_bb6d_80879b1899b2.slice/crio-aa33fbd6976b308aa9fb113ad17a2c11ccd42edb2c23c2ccda6546ce449b1ac7 WatchSource:0}: Error finding container aa33fbd6976b308aa9fb113ad17a2c11ccd42edb2c23c2ccda6546ce449b1ac7: Status 404 returned error can't find the container with id aa33fbd6976b308aa9fb113ad17a2c11ccd42edb2c23c2ccda6546ce449b1ac7
Oct 30 07:15:34.023223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.023200    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7430-e8fc66"]
Oct 30 07:15:34.097237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.097198    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" podUID="25ebf63a-2d38-425c-bf18-15840044abdf" containerName="payload-processing" containerID="cri-o://bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1" gracePeriod=25
Oct 30 07:15:34.131073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.131031    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" podUID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" containerName="payload-pre-processing" containerID="cri-o://8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b" gracePeriod=25
Oct 30 07:15:34.133898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.133865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:34.450544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.450510    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:15:34.471762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.471712    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:34.504481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.504452    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:15:34.539950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.539915    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\" (UniqueName: \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\") pod \"25ebf63a-2d38-425c-bf18-15840044abdf\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") "
Oct 30 07:15:34.540156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.539991    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\" (UniqueName: \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\") pod \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") "
Oct 30 07:15:34.540156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.540111    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\") pod \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\" (UID: \"52d145b9-bc63-4c7d-9acf-04cefa82cb07\") "
Oct 30 07:15:34.540352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.540173    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\") pod \"25ebf63a-2d38-425c-bf18-15840044abdf\" (UID: \"25ebf63a-2d38-425c-bf18-15840044abdf\") "
Oct 30 07:15:34.540756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.540700    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume" pod "52d145b9-bc63-4c7d-9acf-04cefa82cb07" (UID: "52d145b9-bc63-4c7d-9acf-04cefa82cb07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:34.541036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.541011    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume" pod "25ebf63a-2d38-425c-bf18-15840044abdf" (UID: "25ebf63a-2d38-425c-bf18-15840044abdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:34.541110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.541086    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d145b9-bc63-4c7d-9acf-04cefa82cb07-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:34.542795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.542766    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx" pod "25ebf63a-2d38-425c-bf18-15840044abdf" (UID: "25ebf63a-2d38-425c-bf18-15840044abdf"). InnerVolumeSpecName "kube-api-access-wr5jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:34.542889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.542863    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl" pod "52d145b9-bc63-4c7d-9acf-04cefa82cb07" (UID: "52d145b9-bc63-4c7d-9acf-04cefa82cb07"). InnerVolumeSpecName "kube-api-access-2w6dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:34.643093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.643059    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6dl\" (UniqueName: \"kubernetes.io/projected/52d145b9-bc63-4c7d-9acf-04cefa82cb07-kube-api-access-2w6dl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:34.643093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.643090    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ebf63a-2d38-425c-bf18-15840044abdf-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:34.643093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.643099    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5jx\" (UniqueName: \"kubernetes.io/projected/25ebf63a-2d38-425c-bf18-15840044abdf-kube-api-access-wr5jx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:34.677419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.677379    6738 generic.go:356] "Generic (PLEG): container finished" podID="25ebf63a-2d38-425c-bf18-15840044abdf" containerID="bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1" exitCode=0
Oct 30 07:15:34.677652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.677422    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"
Oct 30 07:15:34.677652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.677458    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" event={"ID":"25ebf63a-2d38-425c-bf18-15840044abdf","Type":"ContainerDied","Data":"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"}
Oct 30 07:15:34.677652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.677516    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg" event={"ID":"25ebf63a-2d38-425c-bf18-15840044abdf","Type":"ContainerDied","Data":"49db20e22351f19bf3c330f832c62b8a4fe1dc241c534868ece87973cd0bfaca"}
Oct 30 07:15:34.677652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.677547    6738 scope.go:122] "RemoveContainer" containerID="bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"
Oct 30 07:15:34.679348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.679320    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7430-e8fc66" event={"ID":"49f1933d-5f12-494f-bb6d-80879b1899b2","Type":"ContainerStarted","Data":"aa33fbd6976b308aa9fb113ad17a2c11ccd42edb2c23c2ccda6546ce449b1ac7"}
Oct 30 07:15:34.681776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.681748    6738 generic.go:356] "Generic (PLEG): container finished" podID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" containerID="8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b" exitCode=0
Oct 30 07:15:34.681898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.681807    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"
Oct 30 07:15:34.681898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.681818    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" event={"ID":"52d145b9-bc63-4c7d-9acf-04cefa82cb07","Type":"ContainerDied","Data":"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"}
Oct 30 07:15:34.682016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.681936    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz" event={"ID":"52d145b9-bc63-4c7d-9acf-04cefa82cb07","Type":"ContainerDied","Data":"a06c233377e336ccc0790835ea39e0e2991493eaf6a403eda129cbc9ed19b561"}
Oct 30 07:15:34.694946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.694926    6738 scope.go:122] "RemoveContainer" containerID="bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"
Oct 30 07:15:34.695334 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:34.695233    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1\": container with ID starting with bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1 not found: ID does not exist" containerID="bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"
Oct 30 07:15:34.695334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.695270    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1"} err="failed to get container status \"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1\": rpc error: code = NotFound desc = could not find container \"bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1\": container with ID starting with bc1f67239559530d27cbfbabec8eb18ec4f9cdf793f2f969409da3714ce851d1 not found: ID does not exist"
Oct 30 07:15:34.695334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.695289    6738 scope.go:122] "RemoveContainer" containerID="8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"
Oct 30 07:15:34.708962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.708929    6738 scope.go:122] "RemoveContainer" containerID="8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"
Oct 30 07:15:34.709459 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:34.709426    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b\": container with ID starting with 8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b not found: ID does not exist" containerID="8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"
Oct 30 07:15:34.709587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.709467    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b"} err="failed to get container status \"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b\": rpc error: code = NotFound desc = could not find container \"8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b\": container with ID starting with 8fd64480ded4ca8477d26a90e572f42dddbbe04de23410ff0ee1ddec65a5289b not found: ID does not exist"
Oct 30 07:15:34.738824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.738787    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:15:34.744081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.744048    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f357db-57ff8969d-pzrmg"]
Oct 30 07:15:34.786355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.786282    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:15:34.791213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.791141    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f357db-d779846bc-zrbhz"]
Oct 30 07:15:34.920937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.920906    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:15:34.922001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.921974    6738 cpu_manager.go:414] "containerMap: removing container" podUID="25ebf63a-2d38-425c-bf18-15840044abdf" containerName="payload-processing"
Oct 30 07:15:34.922001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.921996    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ebf63a-2d38-425c-bf18-15840044abdf" containerName="payload-processing"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922045    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922051    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922059    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922064    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922081    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922086    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922099    6738 cpu_manager.go:414] "containerMap: removing container" podUID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" containerName="payload-pre-processing"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922104    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" containerName="payload-pre-processing"
Oct 30 07:15:34.922194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922195    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" containerName="payload-pre-processing"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922206    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ebf63a-2d38-425c-bf18-15840044abdf" containerName="payload-processing"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922214    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="90706db2-1f6c-4280-8277-14a0a3a82afb" containerName="istio-proxy"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922220    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b166cb-9a21-4f88-8e50-ad75cf604db5" containerName="cleanup"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922227    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c770148-a820-4ccb-a852-5f6d5bb500b6" containerName="cleanup"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922236    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac0407-ad88-4786-8165-3128f9465b55" containerName="cleanup"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922243    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce38fb1d-6fd8-46b4-911e-046c062e1a52" containerName="cleanup"
Oct 30 07:15:34.922510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.922249    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0c2f64-d41b-4faf-86cf-e41327022c30" containerName="cleanup"
Oct 30 07:15:34.926687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.926658    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:34.955491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.954664    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-092c5c-openshift-default-dockercfg-tmtvs\""
Oct 30 07:15:34.974509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:34.974473    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:15:35.018085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.018037    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064333    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064398    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064443    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064511    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064645    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67sg\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064689    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064752    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064786    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.067373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.064945    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167555    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167650    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167694    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167733    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167811    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.167991    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67sg\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.168055    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.168124    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.168163    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.169331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.169074    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.170037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.169390    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.170037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.169626    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.173335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.170170    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.185249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.183601    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.194494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.194399    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.243340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.238427    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.243340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.243144    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67sg\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.257515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.257438    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\") pod \"e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") " pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.271338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.270776    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:35.408525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.408465    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:35.693445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.693356    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ebf63a-2d38-425c-bf18-15840044abdf" path="/var/lib/kubelet/pods/25ebf63a-2d38-425c-bf18-15840044abdf/volumes"
Oct 30 07:15:35.694054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.694033    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d145b9-bc63-4c7d-9acf-04cefa82cb07" path="/var/lib/kubelet/pods/52d145b9-bc63-4c7d-9acf-04cefa82cb07/volumes"
Oct 30 07:15:35.724441 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:35.724405    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27b7448_02ca_4d50_9961_1c5c4ef86f38.slice/crio-1ababed8150ab72d3431436e4c4f2e9059f279a0b164f476a74c1d434088a31f WatchSource:0}: Error finding container 1ababed8150ab72d3431436e4c4f2e9059f279a0b164f476a74c1d434088a31f: Status 404 returned error can't find the container with id 1ababed8150ab72d3431436e4c4f2e9059f279a0b164f476a74c1d434088a31f
Oct 30 07:15:35.725427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.725402    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:15:35.727494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.727459    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:35.727609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.727534    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:35.727609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:35.727565    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:36.206629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:36.206592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:36.710326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:36.710279    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerStarted","Data":"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"}
Oct 30 07:15:36.710734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:36.710346    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerStarted","Data":"1ababed8150ab72d3431436e4c4f2e9059f279a0b164f476a74c1d434088a31f"}
Oct 30 07:15:36.782833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:36.782782    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podStartSLOduration=2.782768557 podStartE2EDuration="2.782768557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:36.7690004 +0000 UTC m=+1097.201764294" watchObservedRunningTime="2026-10-30 07:15:36.782768557 +0000 UTC m=+1097.215532450"
Oct 30 07:15:37.031791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.031692    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"]
Oct 30 07:15:37.035961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.035928    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2db9512-qnc2k"]
Oct 30 07:15:37.271996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.271966    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:37.277205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.277180    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:37.518963 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:37.518924    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:15:37.691161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.691128    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a042b0-89f3-4466-9ca4-428fa12ba061" path="/var/lib/kubelet/pods/f1a042b0-89f3-4466-9ca4-428fa12ba061/volumes"
Oct 30 07:15:37.719851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.719816    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7430-e8fc66" event={"ID":"49f1933d-5f12-494f-bb6d-80879b1899b2","Type":"ContainerStarted","Data":"a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872"}
Oct 30 07:15:37.720372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.720277    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:37.721360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.721341    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:15:37.767150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:37.767077    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7430-e8fc66" podStartSLOduration=2.072032451 podStartE2EDuration="4.76705621s" totalImagesPullingTime="2.695023759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:37.758018452 +0000 UTC m=+1098.190782346" watchObservedRunningTime="2026-10-30 07:15:37.76705621 +0000 UTC m=+1098.199820104"
Oct 30 07:15:38.206048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:38.206009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:38.684036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:38.684001    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7430-e8fc66"]
Oct 30 07:15:38.713482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:38.713445    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7430-e8fc66"]
Oct 30 07:15:39.086742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.086661    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"]
Oct 30 07:15:39.087405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.087389    6738 cpu_manager.go:414] "containerMap: removing container" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" containerName="test-curl-7430-e8fc66"
Oct 30 07:15:39.087461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.087406    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" containerName="test-curl-7430-e8fc66"
Oct 30 07:15:39.087527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.087518    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" containerName="test-curl-7430-e8fc66"
Oct 30 07:15:39.092238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.092206    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.110571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.110542    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"]
Oct 30 07:15:39.118824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.118712    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:15:39.118824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.118712    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:15:39.118824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.118712    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:15:39.118824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.118788    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:15:39.148744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.148707    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.148924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.148777    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.148924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.148840    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qp62\" (UniqueName: \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.245659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.245620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:39.250378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.250287    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.250571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.250396    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.250571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.250434    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qp62\" (UniqueName: \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.251160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.251128    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.253007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.252984    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.280220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.280189    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qp62\" (UniqueName: \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\") pod \"maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.402203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.402159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:39.411133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.411104    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:39.724390 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:39.724359    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdde0c7a_969d_4529_a6a7_514322813679.slice/crio-4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127 WatchSource:0}: Error finding container 4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127: Status 404 returned error can't find the container with id 4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127
Oct 30 07:15:39.724886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.724862    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"]
Oct 30 07:15:39.739638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.739605    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp" event={"ID":"fdde0c7a-969d-4529-a6a7-514322813679","Type":"ContainerStarted","Data":"4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127"}
Oct 30 07:15:39.739930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:39.739896    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7430-e8fc66" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" containerName="test-curl-7430-e8fc66" containerID="cri-o://a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872" gracePeriod=2
Oct 30 07:15:40.749856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:40.749823    6738 generic.go:356] "Generic (PLEG): container finished" podID="fdde0c7a-969d-4529-a6a7-514322813679" containerID="b8bdc9699efcc5135ef3c059f0b14353df6940e1169606c5b309c13ac6e831a8" exitCode=0
Oct 30 07:15:40.750284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:40.749910    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp" event={"ID":"fdde0c7a-969d-4529-a6a7-514322813679","Type":"ContainerDied","Data":"b8bdc9699efcc5135ef3c059f0b14353df6940e1169606c5b309c13ac6e831a8"}
Oct 30 07:15:40.986017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:40.985921    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:41.261406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.261318    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:15:41.262080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.262054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:41.265835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.265810    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.291754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.291719    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:15:41.299603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.299571    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-092c5c\""
Oct 30 07:15:41.348210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.348165    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:15:41.352944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.352914    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.370677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.370646    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:15:41.385697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.385654    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.385853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.385713    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dwp\" (UniqueName: \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.385916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.385843    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.386085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.386059    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jk5z\" (UniqueName: \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.388076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.388053    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-092c5c-dockercfg-gntsp\""
Oct 30 07:15:41.497316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.490961    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.497316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.491035    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dwp\" (UniqueName: \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.497316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.491091    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.497316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.491331    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk5z\" (UniqueName: \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.497316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.492758    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.542872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.535330    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.623319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.622008    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jk5z\" (UniqueName: \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\") pod \"payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.679806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.676728    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dwp\" (UniqueName: \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:41.679806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.678214    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:41.883009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:41.882176    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:42.057367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.057263    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:42.093407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.093379    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:42.125317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.115425    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\") pod \"fdde0c7a-969d-4529-a6a7-514322813679\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") "
Oct 30 07:15:42.125317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.115655    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\") pod \"fdde0c7a-969d-4529-a6a7-514322813679\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") "
Oct 30 07:15:42.125317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.115767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\" (UniqueName: \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\") pod \"fdde0c7a-969d-4529-a6a7-514322813679\" (UID: \"fdde0c7a-969d-4529-a6a7-514322813679\") "
Oct 30 07:15:42.125317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.117606    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca" pod "fdde0c7a-969d-4529-a6a7-514322813679" (UID: "fdde0c7a-969d-4529-a6a7-514322813679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:42.139073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.137712    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62" pod "fdde0c7a-969d-4529-a6a7-514322813679" (UID: "fdde0c7a-969d-4529-a6a7-514322813679"). InnerVolumeSpecName "kube-api-access-4qp62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:42.139410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.139360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca" pod "fdde0c7a-969d-4529-a6a7-514322813679" (UID: "fdde0c7a-969d-4529-a6a7-514322813679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:15:42.140732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.140709    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:15:42.142028 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:42.141996    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730c772e_adf0_41d3_88d3_2e428ce56e6c.slice/crio-ddac86ba4ba329f0ff7c37e98e876623a1e5f1c63a347e9c21063c3fe20c6555 WatchSource:0}: Error finding container ddac86ba4ba329f0ff7c37e98e876623a1e5f1c63a347e9c21063c3fe20c6555: Status 404 returned error can't find the container with id ddac86ba4ba329f0ff7c37e98e876623a1e5f1c63a347e9c21063c3fe20c6555
Oct 30 07:15:42.217652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.217620    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\" (UniqueName: \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\") pod \"49f1933d-5f12-494f-bb6d-80879b1899b2\" (UID: \"49f1933d-5f12-494f-bb6d-80879b1899b2\") "
Oct 30 07:15:42.218200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.218180    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:42.218318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.218203    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qp62\" (UniqueName: \"kubernetes.io/projected/fdde0c7a-969d-4529-a6a7-514322813679-kube-api-access-4qp62\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:42.218318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.218219    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdde0c7a-969d-4529-a6a7-514322813679-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:42.219540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.219519    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj" pod "49f1933d-5f12-494f-bb6d-80879b1899b2" (UID: "49f1933d-5f12-494f-bb6d-80879b1899b2"). InnerVolumeSpecName "kube-api-access-p79hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:42.227947 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:42.227921    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057c1400_5914_476d_846b_42dab7eb3613.slice/crio-cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10 WatchSource:0}: Error finding container cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10: Status 404 returned error can't find the container with id cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10
Oct 30 07:15:42.228860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.228835    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:15:42.319423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.319287    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p79hj\" (UniqueName: \"kubernetes.io/projected/49f1933d-5f12-494f-bb6d-80879b1899b2-kube-api-access-p79hj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:42.830981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.830945    6738 generic.go:356] "Generic (PLEG): container finished" podID="49f1933d-5f12-494f-bb6d-80879b1899b2" containerID="a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872" exitCode=137
Oct 30 07:15:42.831149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.830988    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:42.831149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.831038    6738 scope.go:122] "RemoveContainer" containerID="a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872"
Oct 30 07:15:42.833279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.833249    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" event={"ID":"730c772e-adf0-41d3-88d3-2e428ce56e6c","Type":"ContainerStarted","Data":"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"}
Oct 30 07:15:42.833444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.833345    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:42.833444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.833371    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" event={"ID":"730c772e-adf0-41d3-88d3-2e428ce56e6c","Type":"ContainerStarted","Data":"ddac86ba4ba329f0ff7c37e98e876623a1e5f1c63a347e9c21063c3fe20c6555"}
Oct 30 07:15:42.835141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.835111    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" event={"ID":"057c1400-5914-476d-846b-42dab7eb3613","Type":"ContainerStarted","Data":"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"}
Oct 30 07:15:42.835261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.835167    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" event={"ID":"057c1400-5914-476d-846b-42dab7eb3613","Type":"ContainerStarted","Data":"cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10"}
Oct 30 07:15:42.835261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.835198    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:42.836672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.836648    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp" event={"ID":"fdde0c7a-969d-4529-a6a7-514322813679","Type":"ContainerDied","Data":"4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127"}
Oct 30 07:15:42.836767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.836694    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3e2c0f73bcc41248ef453853a2bc84623ffc94d656a2b841b41f3fbd44a127"
Oct 30 07:15:42.836767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.836747    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"
Oct 30 07:15:42.851147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.851116    6738 scope.go:122] "RemoveContainer" containerID="a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872"
Oct 30 07:15:42.851775 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:42.851744    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872\": container with ID starting with a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872 not found: ID does not exist" containerID="a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872"
Oct 30 07:15:42.851880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.851785    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872"} err="failed to get container status \"a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872\": rpc error: code = NotFound desc = could not find container \"a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872\": container with ID starting with a3210ac7a889415cb0162d2c04fd09956340dba0fe3a4faceab7604536b59872 not found: ID does not exist"
Oct 30 07:15:42.878602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.878540    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" podStartSLOduration=1.878521793 podStartE2EDuration="1.878521793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:42.877171474 +0000 UTC m=+1103.309935367" watchObservedRunningTime="2026-10-30 07:15:42.878521793 +0000 UTC m=+1103.311285687"
Oct 30 07:15:42.928780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.928716    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" podStartSLOduration=1.928695409 podStartE2EDuration="1.928695409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:42.921612116 +0000 UTC m=+1103.354376034" watchObservedRunningTime="2026-10-30 07:15:42.928695409 +0000 UTC m=+1103.361459303"
Oct 30 07:15:42.952673 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:42.952627    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7430-e8fc66\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" pod="opendatahub/test-curl-7430-e8fc66"
Oct 30 07:15:42.980968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:42.980934    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:43.864529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:43.864040    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f1933d-5f12-494f-bb6d-80879b1899b2" path="/var/lib/kubelet/pods/49f1933d-5f12-494f-bb6d-80879b1899b2/volumes"
Oct 30 07:15:43.953108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:43.953077    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:44.131397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:44.116163    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:44.131397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:44.124485    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"]
Oct 30 07:15:44.161017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:44.160967    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c7884-fp6ds"]
Oct 30 07:15:44.233263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:44.233222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:44.616998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:44.616917    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:45.031926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:45.031885    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"]
Oct 30 07:15:45.044145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:45.044103    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d62e8a-c4pvj"]
Oct 30 07:15:45.690862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:45.690818    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55804b2b-4b87-4ee2-b7b6-554297b61d4e" path="/var/lib/kubelet/pods/55804b2b-4b87-4ee2-b7b6-554297b61d4e/volumes"
Oct 30 07:15:45.691482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:45.691457    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af3d83f-f4f2-4511-b22f-33247823e9c4" path="/var/lib/kubelet/pods/6af3d83f-f4f2-4511-b22f-33247823e9c4/volumes"
Oct 30 07:15:46.357489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:46.357451    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:47.051318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:47.050781    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"]
Oct 30 07:15:47.064377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:47.064167    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9332cfbe-vx9j9"]
Oct 30 07:15:47.753261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:47.753219    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9765c6b5-6f11-4203-92e7-e5d9fcd64281" path="/var/lib/kubelet/pods/9765c6b5-6f11-4203-92e7-e5d9fcd64281/volumes"
Oct 30 07:15:48.584099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:48.584024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:48.758137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:48.758096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:49.504972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.504933    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"]
Oct 30 07:15:49.505807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.505783    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fdde0c7a-969d-4529-a6a7-514322813679" containerName="revoke-keys"
Oct 30 07:15:49.505807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.505806    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdde0c7a-969d-4529-a6a7-514322813679" containerName="revoke-keys"
Oct 30 07:15:49.506048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.506034    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdde0c7a-969d-4529-a6a7-514322813679" containerName="revoke-keys"
Oct 30 07:15:49.511653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.511628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.535076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.535043    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"]
Oct 30 07:15:49.542823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.542792    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:15:49.543058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.542821    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:15:49.557586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.557552    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:15:49.557743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.557728    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:15:49.648867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.648824    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vmt\" (UniqueName: \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.649106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.649082    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.649308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.649263    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.750623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.750585    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.750832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.750689    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.750832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.750792    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vmt\" (UniqueName: \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.751386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.751361    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.753094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.753064    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.797243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.797163    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vmt\" (UniqueName: \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\") pod \"maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.827017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.826977    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:49.940555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:49.940518    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:15:50.092074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:50.091922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:50.165562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:50.165533    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"]
Oct 30 07:15:50.166018 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:50.165993    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab4ea99_32cb_47f0_afa4_b1cc2b0ae442.slice/crio-cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d WatchSource:0}: Error finding container cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d: Status 404 returned error can't find the container with id cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d
Oct 30 07:15:50.288365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:50.274898    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:51.158860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.158815    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq" event={"ID":"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442","Type":"ContainerStarted","Data":"371d53bed1013879ecc340eb92edbfab1e49bd1b6de331dae0595e7b8ebda504"}
Oct 30 07:15:51.159256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.158879    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq" event={"ID":"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442","Type":"ContainerStarted","Data":"cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d"}
Oct 30 07:15:51.364479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.348830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:51.500727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.495205    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:51.524597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.522639    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq" podStartSLOduration=2.522620142 podStartE2EDuration="2.522620142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:51.254245622 +0000 UTC m=+1111.687009515" watchObservedRunningTime="2026-10-30 07:15:51.522620142 +0000 UTC m=+1111.955384035"
Oct 30 07:15:51.552334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.547981    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:15:51.552334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.548364    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy" containerID="cri-o://797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310" gracePeriod=30
Oct 30 07:15:51.752341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:51.650579    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:52.227263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:52.227212    6738 generic.go:356] "Generic (PLEG): container finished" podID="9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" containerID="371d53bed1013879ecc340eb92edbfab1e49bd1b6de331dae0595e7b8ebda504" exitCode=0
Oct 30 07:15:52.227263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:52.227264    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq" event={"ID":"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442","Type":"ContainerDied","Data":"371d53bed1013879ecc340eb92edbfab1e49bd1b6de331dae0595e7b8ebda504"}
Oct 30 07:15:53.352830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.352780    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:53.450395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.450366    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:53.508306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.508212    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\" (UniqueName: \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\") pod \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") "
Oct 30 07:15:53.508306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.508285    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\") pod \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") "
Oct 30 07:15:53.508551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.508482    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\") pod \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\" (UID: \"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442\") "
Oct 30 07:15:53.510006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.509964    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca" pod "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" (UID: "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:53.511004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.510973    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca" pod "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" (UID: "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:15:53.511984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.511939    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt" pod "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" (UID: "9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442"). InnerVolumeSpecName "kube-api-access-b9vmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:53.610794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.610760    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vmt\" (UniqueName: \"kubernetes.io/projected/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-kube-api-access-b9vmt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:53.610794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.610791    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:53.610794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.610800    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:53.708478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.708443    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:53.801691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.801601    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:15:53.802342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.802286    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" containerName="revoke-keys"
Oct 30 07:15:53.802446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.802347    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" containerName="revoke-keys"
Oct 30 07:15:53.802548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.802537    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" containerName="revoke-keys"
Oct 30 07:15:53.809472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.809445    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.829790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.829755    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:15:53.829956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.829858    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ba67d8-openshift-default-dockercfg-r2cwg\""
Oct 30 07:15:53.890440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.867728    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:53.912349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.907229    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:15:53.912349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.911068    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.987954    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988061    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988100    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988165    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988309    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988334    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:53.990130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988422    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.015800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:53.988458    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vs9\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.015800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.014962    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.077161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.076886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119321    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119385    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119477    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119610    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119637    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119738    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119777    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vs9\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119867    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.119975    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.120527    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.121236    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.158533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.121508    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.162544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.159679    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.162544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.162070    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.261450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.218374    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.369472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.283921    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vs9\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.369472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.318094    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.410454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.381140    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.433463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.433427    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:54.551342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.545049    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq" event={"ID":"9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442","Type":"ContainerDied","Data":"cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d"}
Oct 30 07:15:54.551342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.545131    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb8fb75ddcd81aab72e52eda97111eae3b0ddd9ef29620437f352a7138ead8d"
Oct 30 07:15:54.551342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.545378    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"
Oct 30 07:15:54.912710 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:54.912676    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4cbf02_a776_4090_8bba_c1d6655028d5.slice/crio-bc7d8f1c5f8b05dcf953eb602c31497f218b9b4f93e7215905dff9f033e5dd4b WatchSource:0}: Error finding container bc7d8f1c5f8b05dcf953eb602c31497f218b9b4f93e7215905dff9f033e5dd4b: Status 404 returned error can't find the container with id bc7d8f1c5f8b05dcf953eb602c31497f218b9b4f93e7215905dff9f033e5dd4b
Oct 30 07:15:54.913144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.913118    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:15:54.915472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.915434    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:54.915582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.915506    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:54.915582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.915534    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:54.941760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:54.941074    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerName="payload-processing" containerID="cri-o://ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459" gracePeriod=25
Oct 30 07:15:55.232120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.232084    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:15:55.265039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.265002    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\" (UniqueName: \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\") pod \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") "
Oct 30 07:15:55.265219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.265101    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\") pod \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\" (UID: \"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594\") "
Oct 30 07:15:55.265531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.265495    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume" pod "3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" (UID: "3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:55.265793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.265773    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:55.267353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.267289    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg" pod "3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" (UID: "3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594"). InnerVolumeSpecName "kube-api-access-hbjsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:55.366841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.366787    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjsg\" (UniqueName: \"kubernetes.io/projected/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594-kube-api-access-hbjsg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:55.556597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.556506    6738 generic.go:356] "Generic (PLEG): container finished" podID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerID="ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459" exitCode=0
Oct 30 07:15:55.556597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.556543    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" event={"ID":"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594","Type":"ContainerDied","Data":"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"}
Oct 30 07:15:55.556597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.556579    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"
Oct 30 07:15:55.557154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.556613    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" event={"ID":"3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594","Type":"ContainerDied","Data":"f64bc302f2cc229faff2fd496f70471ad06be2407162a71cd15300314ac829f5"}
Oct 30 07:15:55.557154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.556645    6738 scope.go:122] "RemoveContainer" containerID="ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"
Oct 30 07:15:55.559325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.559258    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerStarted","Data":"f372e1c8d8d4067add08641451a75dc19973043aa94bbe2ae500d67a4226393e"}
Oct 30 07:15:55.559439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.559409    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerStarted","Data":"bc7d8f1c5f8b05dcf953eb602c31497f218b9b4f93e7215905dff9f033e5dd4b"}
Oct 30 07:15:55.572540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.572513    6738 scope.go:122] "RemoveContainer" containerID="ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"
Oct 30 07:15:55.572921 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:55.572900    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459\": container with ID starting with ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459 not found: ID does not exist" containerID="ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"
Oct 30 07:15:55.572994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.572932    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459"} err="failed to get container status \"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459\": rpc error: code = NotFound desc = could not find container \"ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459\": container with ID starting with ee88793cad1fbd15a61b89b2369244d1878eaf7b8d8f58591db717bc425e1459 not found: ID does not exist"
Oct 30 07:15:55.614949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.614911    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:15:55.620358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.620329    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z"]
Oct 30 07:15:55.641479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.641441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:55.673039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.672976    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podStartSLOduration=2.672958292 podStartE2EDuration="2.672958292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:55.667145029 +0000 UTC m=+1116.099908916" watchObservedRunningTime="2026-10-30 07:15:55.672958292 +0000 UTC m=+1116.105722187"
Oct 30 07:15:55.810413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:55.802699    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" path="/var/lib/kubelet/pods/3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594/volumes"
Oct 30 07:15:56.130877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.130831    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a6a14e07-f46d6798b-vpd9z" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.131:50052\" within 1s: context deadline exceeded"
Oct 30 07:15:56.131059 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:15:56.130958    6738 logging.go:55] [core] [Channel #1270 SubChannel #1271]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.131:50052", ServerName: "10.133.0.131:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.131:50052: operation was canceled"
Oct 30 07:15:56.415147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.415052    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:56.436108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.436069    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:56.442776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.442743    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:56.570558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.570525    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:56.572446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:56.572416    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:15:57.339530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.339503    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:15:57.434475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434386    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434494    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434613    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434645    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434758    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434814    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.434927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.434864    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:57.435040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435023    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.435106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435059    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.435156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435114    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\") pod \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\" (UID: \"2d4264b8-a46c-4057-8fcb-65c23d1f1568\") "
Oct 30 07:15:57.435265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435239    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:57.435357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435255    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:57.435461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435438    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:57.435626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.435600    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:57.438813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.438782    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.438813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.438815    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.438993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.438832    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.438993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.438857    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.438993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.438870    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.444648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.443405    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:15:57.444648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.443678    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:15:57.444648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.443796    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "kube-api-access-fcgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:57.452204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.452169    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token" pod "2d4264b8-a46c-4057-8fcb-65c23d1f1568" (UID: "2d4264b8-a46c-4057-8fcb-65c23d1f1568"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:57.540515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.540472    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.540515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.540515    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.540733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.540530    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.540733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.540542    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgwz\" (UniqueName: \"kubernetes.io/projected/2d4264b8-a46c-4057-8fcb-65c23d1f1568-kube-api-access-fcgwz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:15:57.580740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.580704    6738 generic.go:356] "Generic (PLEG): container finished" podID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerID="797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310" exitCode=0
Oct 30 07:15:57.581231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.580770    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"
Oct 30 07:15:57.581231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.580859    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerDied","Data":"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"}
Oct 30 07:15:57.581231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.580943    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf" event={"ID":"2d4264b8-a46c-4057-8fcb-65c23d1f1568","Type":"ContainerDied","Data":"f3e37ae6369a96dd4d87fc3bfa4aad0f8d26196242dc2c628f3433459a42e782"}
Oct 30 07:15:57.581231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.580977    6738 scope.go:122] "RemoveContainer" containerID="797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"
Oct 30 07:15:57.594097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.594074    6738 scope.go:122] "RemoveContainer" containerID="ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"
Oct 30 07:15:57.605646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.605622    6738 scope.go:122] "RemoveContainer" containerID="797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"
Oct 30 07:15:57.605993 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:57.605964    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310\": container with ID starting with 797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310 not found: ID does not exist" containerID="797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"
Oct 30 07:15:57.606121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.605998    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310"} err="failed to get container status \"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310\": rpc error: code = NotFound desc = could not find container \"797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310\": container with ID starting with 797188778bb3dedeecf334a1769bd21db5e3eeea979361766a5ba960bc73c310 not found: ID does not exist"
Oct 30 07:15:57.606121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.606016    6738 scope.go:122] "RemoveContainer" containerID="ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"
Oct 30 07:15:57.606402 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:15:57.606377    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8\": container with ID starting with ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8 not found: ID does not exist" containerID="ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"
Oct 30 07:15:57.606450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.606411    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8"} err="failed to get container status \"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8\": rpc error: code = NotFound desc = could not find container \"ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8\": container with ID starting with ea0f46c723b91ad192bbaf9e0dca97bc17ebe306ea061e1c2e113a2e6a94c2f8 not found: ID does not exist"
Oct 30 07:15:57.644098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.644055    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:15:57.648258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.648227    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7c7884-openshift-default-7c69c8679c-sqmlf"]
Oct 30 07:15:57.664655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.664620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:57.691330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:57.691240    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" path="/var/lib/kubelet/pods/2d4264b8-a46c-4057-8fcb-65c23d1f1568/volumes"
Oct 30 07:15:58.062365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:58.062263    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"]
Oct 30 07:15:58.063832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:58.063788    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:58.071361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:58.071327    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d24f105-jwhlr"]
Oct 30 07:15:58.218877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:58.218842    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:15:59.601953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.601918    6738 generic.go:356] "Generic (PLEG): container finished" podID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerID="2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b" exitCode=0
Oct 30 07:15:59.602462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.601989    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerDied","Data":"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"}
Oct 30 07:15:59.602528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.602508    6738 scope.go:122] "RemoveContainer" containerID="2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"
Oct 30 07:15:59.604153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.604126    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:59.604228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.604214    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:59.604270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.604248    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:15:59.691916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.691879    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b34d8a6-bbc7-4bab-82ed-8c475ca753fe" path="/var/lib/kubelet/pods/9b34d8a6-bbc7-4bab-82ed-8c475ca753fe/volumes"
Oct 30 07:15:59.949734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:15:59.949697    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:00.038197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038163    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:00.038863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038848    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerName="payload-processing"
Oct 30 07:16:00.038923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038864    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerName="payload-processing"
Oct 30 07:16:00.038923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038891    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.038923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038896    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.038923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038906    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.038923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.038912    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.039097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.039009    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.039097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.039025    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0cee4d-8344-44bd-a2e8-a5fbdc2c1594" containerName="payload-processing"
Oct 30 07:16:00.042578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.042556    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.071029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.070803    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba67d8\""
Oct 30 07:16:00.080392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.078907    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"]
Oct 30 07:16:00.087361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.085712    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:00.089334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.089281    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69024c76-lxd8g"]
Oct 30 07:16:00.093378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.093346    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:00.094207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.094188    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4264b8-a46c-4057-8fcb-65c23d1f1568" containerName="istio-proxy"
Oct 30 07:16:00.098337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.098310    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.118357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.107415    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:00.118357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.108470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:00.118357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.114104    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba67d8-dockercfg-m22rf\""
Oct 30 07:16:00.215719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.215620    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp77\" (UniqueName: \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.242731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.242686    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.277472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.277428    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77jj\" (UniqueName: \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.285323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.277809    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.501374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.497846    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.535482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.535431    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77jj\" (UniqueName: \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.535684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.535563    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.535684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.535667    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp77\" (UniqueName: \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.536216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.512401    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.537280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.537253    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.568360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.568321    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77jj\" (UniqueName: \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:00.579317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.573333    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp77\" (UniqueName: \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\") pod \"payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.616868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.616836    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerStarted","Data":"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"}
Oct 30 07:16:00.680110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.678569    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:00.698550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.698514    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:16:00.699869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.699836    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:00.700036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.699913    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:00.748345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:00.748040    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:01.013444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.013413    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:01.014528 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:01.014503    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ad595c_4adc_4bee_b0c7_819ade9c61f7.slice/crio-e99170be9c0d009595a7734ea967388e121b979e5f5de7d999449ce067c5d044 WatchSource:0}: Error finding container e99170be9c0d009595a7734ea967388e121b979e5f5de7d999449ce067c5d044: Status 404 returned error can't find the container with id e99170be9c0d009595a7734ea967388e121b979e5f5de7d999449ce067c5d044
Oct 30 07:16:01.155165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.155131    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:01.254870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.254840    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:01.402737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.402693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:01.628695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.628657    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" event={"ID":"c7120dc5-da24-4ddc-972e-bfb842a814c9","Type":"ContainerStarted","Data":"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"}
Oct 30 07:16:01.629187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.628745    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" event={"ID":"c7120dc5-da24-4ddc-972e-bfb842a814c9","Type":"ContainerStarted","Data":"53c0d40c4f8247e4c60c7c977a9fa6c35f67ff4693a2adafb77233f10ddc7fe5"}
Oct 30 07:16:01.629187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.628773    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:01.630987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.630948    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" event={"ID":"77ad595c-4adc-4bee-b0c7-819ade9c61f7","Type":"ContainerStarted","Data":"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"}
Oct 30 07:16:01.631131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.631009    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" event={"ID":"77ad595c-4adc-4bee-b0c7-819ade9c61f7","Type":"ContainerStarted","Data":"e99170be9c0d009595a7734ea967388e121b979e5f5de7d999449ce067c5d044"}
Oct 30 07:16:01.631205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.631127    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:01.685319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.683837    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" podStartSLOduration=1.6838156309999999 podStartE2EDuration="1.683815631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:01.677855049 +0000 UTC m=+1122.110618946" watchObservedRunningTime="2026-10-30 07:16:01.683815631 +0000 UTC m=+1122.116579525"
Oct 30 07:16:01.690330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.689723    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:01.690330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.689796    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:01.707033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.707002    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e157f459-7711-4935-9143-0e44b7f386e3" path="/var/lib/kubelet/pods/e157f459-7711-4935-9143-0e44b7f386e3/volumes"
Oct 30 07:16:01.721594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:01.721534    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" podStartSLOduration=2.721518706 podStartE2EDuration="2.721518706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:01.718200205 +0000 UTC m=+1122.150964099" watchObservedRunningTime="2026-10-30 07:16:01.721518706 +0000 UTC m=+1122.154282599"
Oct 30 07:16:02.211228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:02.211076    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:02.495400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:02.484777    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:02.692852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:02.692808    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:02.693412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:02.692879    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:03.716320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:03.710317    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:03.716320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:03.710392    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:04.506772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:04.506735    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:04.691331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:04.690730    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:04.691331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:04.690835    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:05.693321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:05.690373    6738 patch_prober.go:29] interesting pod/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 07:16:05.693321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:05.690438    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 07:16:05.755484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:05.755437    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:16:06.695517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:06.695473    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:16:06.726550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:06.726519    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:16:06.934615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:06.934574    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:07.519730 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:07.519694    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:16:08.520240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:08.520195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:09.202247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.202211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:09.576111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.576015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:09.604388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.604347    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"]
Oct 30 07:16:09.612605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.612570    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f357db-tn2rn"]
Oct 30 07:16:09.692130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.692095    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19" path="/var/lib/kubelet/pods/b5e64a1d-dc5a-4128-a44b-ce5fc3d14b19/volumes"
Oct 30 07:16:09.754094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:09.754034    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:10.181541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:10.181507    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:11.060571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:11.060537    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"]
Oct 30 07:16:11.072687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:11.072655    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e6b031-sb4v4"]
Oct 30 07:16:11.691430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:11.691394    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d524e8-32b2-4ec4-a763-e11263546f34" path="/var/lib/kubelet/pods/55d524e8-32b2-4ec4-a763-e11263546f34/volumes"
Oct 30 07:16:11.985931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:11.985844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:12.641539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:12.641509    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:12.641931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:12.641586    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:13.591245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:13.591210    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:14.557995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:14.557958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:14.699443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:14.699410    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:14.721608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:14.721560    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"]
Oct 30 07:16:14.767163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:14.767121    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a6a14e07-npjrp"]
Oct 30 07:16:14.816256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:14.816163    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:15.219556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:15.219519    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:15.693307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:15.693264    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdde0c7a-969d-4529-a6a7-514322813679" path="/var/lib/kubelet/pods/fdde0c7a-969d-4529-a6a7-514322813679/volumes"
Oct 30 07:16:16.574324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:16.574242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:16.698316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:16.694981    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:16.991374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:16.991331    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:17.141341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:17.139455    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:18.011488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.011453    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:18.031092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.031055    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"]
Oct 30 07:16:18.042452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.042421    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9152df1e-8gmwq"]
Oct 30 07:16:18.486650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.486611    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:16:18.501687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.501651    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.504915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.504884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:16:18.517793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.517762    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-137ec6-openshift-default-dockercfg-45qtw\""
Oct 30 07:16:18.548457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.548422    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:18.590790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.590750    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.590804    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.590910    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.590952    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.591167    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.591200    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.591229    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzd8\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.591327    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.591385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.591351    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.666141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.666007    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:18.693430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.693387    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.693729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.693705    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.693909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.693892    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.694025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694011    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.694324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.694474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694456    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.694594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694579    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzd8\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.694711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694698    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.700330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.694797    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.700756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.700135    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.700939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.700427    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.701435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.701410    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.701893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.701621    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.702060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.701812    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.718718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.718683    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.719645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.719615    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.728956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.728857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.741450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.740937    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzd8\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\") pod \"e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") " pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:18.832953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:18.832763    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:19.314500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.314470    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:16:19.314890 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:19.314770    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6c24f1_a0b3_466d_954e_2db6d985cf02.slice/crio-520d4bf5f1f3e6d4fdc960069a7a5b84f110f5d361e6165ef4ff1c65b741aef1 WatchSource:0}: Error finding container 520d4bf5f1f3e6d4fdc960069a7a5b84f110f5d361e6165ef4ff1c65b741aef1: Status 404 returned error can't find the container with id 520d4bf5f1f3e6d4fdc960069a7a5b84f110f5d361e6165ef4ff1c65b741aef1
Oct 30 07:16:19.317688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.317649    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:19.317788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.317722    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:19.317788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.317750    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:19.327641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.327599    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:19.477731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.471179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:19.694091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.694058    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8287013d-a683-4e0f-b786-8e90b46053de" path="/var/lib/kubelet/pods/8287013d-a683-4e0f-b786-8e90b46053de/volumes"
Oct 30 07:16:19.979393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.979269    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerStarted","Data":"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"}
Oct 30 07:16:19.979393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:19.979373    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerStarted","Data":"520d4bf5f1f3e6d4fdc960069a7a5b84f110f5d361e6165ef4ff1c65b741aef1"}
Oct 30 07:16:20.040975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.040924    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podStartSLOduration=2.040908514 podStartE2EDuration="2.040908514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:20.036794487 +0000 UTC m=+1140.469558379" watchObservedRunningTime="2026-10-30 07:16:20.040908514 +0000 UTC m=+1140.473672406"
Oct 30 07:16:20.265373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.258536    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:20.851317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.838023    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:20.875357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.873069    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:20.990552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.990509    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:20.991454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:20.991431    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:16:21.471150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:21.471105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:22.598878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:22.598828    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:22.723636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:22.723601    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:23.051940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:23.051904    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"]
Oct 30 07:16:23.056331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:23.056275    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31722c-9dsgx"]
Oct 30 07:16:23.704635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:23.704596    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:23.727570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:23.727523    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20807347-b99f-4f72-8899-ecdf8002630a" path="/var/lib/kubelet/pods/20807347-b99f-4f72-8899-ecdf8002630a/volumes"
Oct 30 07:16:24.365908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.365873    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:24.489584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.489549    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:24.544438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.544398    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:16:24.551260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.551235    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:24.572849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.572817    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:16:24.579004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.578968    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-137ec6\""
Oct 30 07:16:24.579166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.579137    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-137ec6-dockercfg-z46bk\""
Oct 30 07:16:24.641506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.636759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:24.641506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.636824    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vs6\" (UniqueName: \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:24.960004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.959899    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:24.964019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.963519    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vs6\" (UniqueName: \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:24.970623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:24.970568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:25.016400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.016286    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:16:25.027653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.027617    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vs6\" (UniqueName: \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:25.028968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.028785    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.035779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.035746    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:16:25.079836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.079804    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.080031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.079990    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrv6\" (UniqueName: \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.166538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.166505    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:25.181851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.181820    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.181990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.181960    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrv6\" (UniqueName: \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.182685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.182635    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.205220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.205189    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrv6\" (UniqueName: \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\") pod \"payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.351559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.351521    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:25.487522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.487421    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:16:25.493616 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:25.493582    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f855cb_7fd2_44f8_9ebd_a312aef1814d.slice/crio-f9bc5844a0380d624d7d18cd915c6796c22cf1e43d2a199a0f302f3b775a3741 WatchSource:0}: Error finding container f9bc5844a0380d624d7d18cd915c6796c22cf1e43d2a199a0f302f3b775a3741: Status 404 returned error can't find the container with id f9bc5844a0380d624d7d18cd915c6796c22cf1e43d2a199a0f302f3b775a3741
Oct 30 07:16:25.662135 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:25.662104    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65b93c_f27a_4fd4_a142_8219f7a59a76.slice/crio-b831c3596f0648ad9b4751b507c8284be7c353ff3173af6bcebb387974f7a0f8 WatchSource:0}: Error finding container b831c3596f0648ad9b4751b507c8284be7c353ff3173af6bcebb387974f7a0f8: Status 404 returned error can't find the container with id b831c3596f0648ad9b4751b507c8284be7c353ff3173af6bcebb387974f7a0f8
Oct 30 07:16:25.662810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.662783    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:16:25.666739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:25.666712    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:26.049376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.049340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" event={"ID":"ae65b93c-f27a-4fd4-a142-8219f7a59a76","Type":"ContainerStarted","Data":"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"}
Oct 30 07:16:26.049827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.049395    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" event={"ID":"ae65b93c-f27a-4fd4-a142-8219f7a59a76","Type":"ContainerStarted","Data":"b831c3596f0648ad9b4751b507c8284be7c353ff3173af6bcebb387974f7a0f8"}
Oct 30 07:16:26.049827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.049422    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:26.051196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.051165    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" event={"ID":"b8f855cb-7fd2-44f8-9ebd-a312aef1814d","Type":"ContainerStarted","Data":"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"}
Oct 30 07:16:26.051401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.051209    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" event={"ID":"b8f855cb-7fd2-44f8-9ebd-a312aef1814d","Type":"ContainerStarted","Data":"f9bc5844a0380d624d7d18cd915c6796c22cf1e43d2a199a0f302f3b775a3741"}
Oct 30 07:16:26.051401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.051252    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:26.092817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.092696    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" podStartSLOduration=2.092680025 podStartE2EDuration="2.092680025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:26.089840361 +0000 UTC m=+1146.522604254" watchObservedRunningTime="2026-10-30 07:16:26.092680025 +0000 UTC m=+1146.525443919"
Oct 30 07:16:26.134510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:26.134457    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" podStartSLOduration=2.134443666 podStartE2EDuration="2.134443666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:26.13050365 +0000 UTC m=+1146.563267543" watchObservedRunningTime="2026-10-30 07:16:26.134443666 +0000 UTC m=+1146.567207559"
Oct 30 07:16:28.152561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:28.152522    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:28.345421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:28.339874    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:29.427769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:29.427731    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:30.031758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.031721    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"]
Oct 30 07:16:30.044159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.044122    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265191bd-gmfv6"]
Oct 30 07:16:30.201669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.201622    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:16:30.237873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.237838    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:16:30.348082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.347984    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:30.520359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.517402    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:30.767227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.745459    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:30.908321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:30.903856    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:31.700864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:31.700828    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db96ea6-8f2e-4f8b-b19e-8a1e86155a93" path="/var/lib/kubelet/pods/0db96ea6-8f2e-4f8b-b19e-8a1e86155a93/volumes"
Oct 30 07:16:32.031507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:32.031425    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"]
Oct 30 07:16:32.043860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:32.043826    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92defb9e-gcl6b"]
Oct 30 07:16:32.932787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:32.932750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:33.083317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:33.079734    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"]
Oct 30 07:16:33.083317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:33.082802    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06163e03-jqp2c"]
Oct 30 07:16:33.691214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:33.691182    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1d9080-84a8-4f99-a935-17a1fefae0bb" path="/var/lib/kubelet/pods/1c1d9080-84a8-4f99-a935-17a1fefae0bb/volumes"
Oct 30 07:16:33.691709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:33.691695    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f86e2d-7f2c-477d-af3b-c60781b10626" path="/var/lib/kubelet/pods/e1f86e2d-7f2c-477d-af3b-c60781b10626/volumes"
Oct 30 07:16:34.853902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:34.853864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:35.202380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.202264    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" podUID="730c772e-adf0-41d3-88d3-2e428ce56e6c" containerName="payload-processing" containerID="cri-o://0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6" gracePeriod=25
Oct 30 07:16:35.238392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.238345    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" podUID="057c1400-5914-476d-846b-42dab7eb3613" containerName="payload-pre-processing" containerID="cri-o://8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853" gracePeriod=25
Oct 30 07:16:35.585905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.585868    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:16:35.622512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.622479    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:16:35.664882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.664838    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\" (UniqueName: \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\") pod \"057c1400-5914-476d-846b-42dab7eb3613\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") "
Oct 30 07:16:35.665086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.664908    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\" (UniqueName: \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\") pod \"730c772e-adf0-41d3-88d3-2e428ce56e6c\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") "
Oct 30 07:16:35.679323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.670957    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z" pod "730c772e-adf0-41d3-88d3-2e428ce56e6c" (UID: "730c772e-adf0-41d3-88d3-2e428ce56e6c"). InnerVolumeSpecName "kube-api-access-8jk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:35.679323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.676747    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp" pod "057c1400-5914-476d-846b-42dab7eb3613" (UID: "057c1400-5914-476d-846b-42dab7eb3613"). InnerVolumeSpecName "kube-api-access-j2dwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:35.767679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.767585    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\") pod \"057c1400-5914-476d-846b-42dab7eb3613\" (UID: \"057c1400-5914-476d-846b-42dab7eb3613\") "
Oct 30 07:16:35.767679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.767632    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\") pod \"730c772e-adf0-41d3-88d3-2e428ce56e6c\" (UID: \"730c772e-adf0-41d3-88d3-2e428ce56e6c\") "
Oct 30 07:16:35.768030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768005    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume" pod "057c1400-5914-476d-846b-42dab7eb3613" (UID: "057c1400-5914-476d-846b-42dab7eb3613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:35.768121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768093    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume" pod "730c772e-adf0-41d3-88d3-2e428ce56e6c" (UID: "730c772e-adf0-41d3-88d3-2e428ce56e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:35.768655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768622    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057c1400-5914-476d-846b-42dab7eb3613-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.768655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768652    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c772e-adf0-41d3-88d3-2e428ce56e6c-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.768830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768764    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dwp\" (UniqueName: \"kubernetes.io/projected/057c1400-5914-476d-846b-42dab7eb3613-kube-api-access-j2dwp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.768830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:35.768781    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jk5z\" (UniqueName: \"kubernetes.io/projected/730c772e-adf0-41d3-88d3-2e428ce56e6c-kube-api-access-8jk5z\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:36.246900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.246864    6738 generic.go:356] "Generic (PLEG): container finished" podID="730c772e-adf0-41d3-88d3-2e428ce56e6c" containerID="0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6" exitCode=0
Oct 30 07:16:36.247429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.246988    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"
Oct 30 07:16:36.247429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.247006    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" event={"ID":"730c772e-adf0-41d3-88d3-2e428ce56e6c","Type":"ContainerDied","Data":"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"}
Oct 30 07:16:36.247429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.247081    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt" event={"ID":"730c772e-adf0-41d3-88d3-2e428ce56e6c","Type":"ContainerDied","Data":"ddac86ba4ba329f0ff7c37e98e876623a1e5f1c63a347e9c21063c3fe20c6555"}
Oct 30 07:16:36.247429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.247116    6738 scope.go:122] "RemoveContainer" containerID="0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"
Oct 30 07:16:36.251218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.251180    6738 generic.go:356] "Generic (PLEG): container finished" podID="057c1400-5914-476d-846b-42dab7eb3613" containerID="8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853" exitCode=0
Oct 30 07:16:36.251465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.251261    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"
Oct 30 07:16:36.251605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.251280    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" event={"ID":"057c1400-5914-476d-846b-42dab7eb3613","Type":"ContainerDied","Data":"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"}
Oct 30 07:16:36.251675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.251618    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd" event={"ID":"057c1400-5914-476d-846b-42dab7eb3613","Type":"ContainerDied","Data":"cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10"}
Oct 30 07:16:36.265045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.265022    6738 scope.go:122] "RemoveContainer" containerID="0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"
Oct 30 07:16:36.265432 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:36.265409    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6\": container with ID starting with 0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6 not found: ID does not exist" containerID="0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"
Oct 30 07:16:36.265533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.265444    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6"} err="failed to get container status \"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6\": rpc error: code = NotFound desc = could not find container \"0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6\": container with ID starting with 0acacff6ffb3a61aedb6d8988ec59abdcac7dafc762aafea992f5d88e37a53b6 not found: ID does not exist"
Oct 30 07:16:36.265533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.265469    6738 scope.go:122] "RemoveContainer" containerID="8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"
Oct 30 07:16:36.277176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.277148    6738 scope.go:122] "RemoveContainer" containerID="8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"
Oct 30 07:16:36.277592 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:36.277569    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853\": container with ID starting with 8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853 not found: ID does not exist" containerID="8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"
Oct 30 07:16:36.277692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.277598    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853"} err="failed to get container status \"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853\": rpc error: code = NotFound desc = could not find container \"8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853\": container with ID starting with 8d032f0f928496286023bad3062bea000727d5ca1c4152ddb11cdb0af1780853 not found: ID does not exist"
Oct 30 07:16:36.326081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.326028    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:16:36.327794 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:36.327750    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730c772e_adf0_41d3_88d3_2e428ce56e6c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057c1400_5914_476d_846b_42dab7eb3613.slice/crio-cbebf53704af776061deeb74efd7e28d3314c1a518200b954bb5c0ad47dcdc10\": RecentStats: unable to find data in memory cache]"
Oct 30 07:16:36.337329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.337287    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-576db44f5-c9nhd"]
Oct 30 07:16:36.365118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.365075    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:16:36.369002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.368971    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-56bf5c658b-46lmt"]
Oct 30 07:16:36.744876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:36.744837    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:37.118373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.118258    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:16:37.118533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.118376    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:16:37.520607 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:37.520572    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:16:37.691275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.691234    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057c1400-5914-476d-846b-42dab7eb3613" path="/var/lib/kubelet/pods/057c1400-5914-476d-846b-42dab7eb3613/volumes"
Oct 30 07:16:37.691874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.691851    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730c772e-adf0-41d3-88d3-2e428ce56e6c" path="/var/lib/kubelet/pods/730c772e-adf0-41d3-88d3-2e428ce56e6c/volumes"
Oct 30 07:16:37.722354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.722263    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:37.722505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:37.722363    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:38.273413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.273372    6738 generic.go:356] "Generic (PLEG): container finished" podID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerID="45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066" exitCode=0
Oct 30 07:16:38.273610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.273428    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerDied","Data":"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"}
Oct 30 07:16:38.273913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.273898    6738 scope.go:122] "RemoveContainer" containerID="45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"
Oct 30 07:16:38.275602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.275573    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:38.275688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.275651    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:38.275727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.275689    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:38.710477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:38.710441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:39.298846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:39.298810    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerStarted","Data":"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"}
Oct 30 07:16:40.034436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.034404    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"]
Oct 30 07:16:40.046472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.046441    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7ef020-fddp6"]
Oct 30 07:16:40.287669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.287587    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:16:40.288030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.288002    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:40.288208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.288181    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:40.356361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:40.356319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:41.117319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.099983    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:41.272763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.272724    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:41.273098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.273058    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:41.434253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.434218    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:41.477432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.477378    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"]
Oct 30 07:16:41.501464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.501415    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0e3de-vd9lq"]
Oct 30 07:16:41.587935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.587898    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:41.748538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.748455    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c62b42d-e9f6-418a-a831-f8840f148e69" path="/var/lib/kubelet/pods/7c62b42d-e9f6-418a-a831-f8840f148e69/volumes"
Oct 30 07:16:41.750747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:41.750716    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442" path="/var/lib/kubelet/pods/9ab4ea99-32cb-47f0-afa4-b1cc2b0ae442/volumes"
Oct 30 07:16:42.020056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:42.019970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:42.036045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:42.036008    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"]
Oct 30 07:16:42.047978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:42.047948    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c39a6ed4-zhs2r"]
Oct 30 07:16:42.288379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:42.280730    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:42.288379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:42.280795    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:43.273803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:43.273760    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:43.274121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:43.274090    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:43.758667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:43.758476    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7712f1-8b1f-40fd-912b-166cffc6084f" path="/var/lib/kubelet/pods/5a7712f1-8b1f-40fd-912b-166cffc6084f/volumes"
Oct 30 07:16:44.129432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:44.129395    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:44.253782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:44.253747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:44.276784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:44.276740    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 07:16:44.277091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:44.277057    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 07:16:44.421439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:44.421283    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:45.276388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:45.276358    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:16:45.276865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:45.276838    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:16:45.277681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:45.277634    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:16:46.332829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:46.332775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:48.167497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.167455    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:48.296171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.296105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:48.474401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.474310    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:16:48.474576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.474531    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy" containerID="cri-o://3daff7c6a342345b4cdd4aaa6f11f6a6d1be586be893f91a18102c3f19469b88" gracePeriod=30
Oct 30 07:16:48.507402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.507356    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:48.760128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:48.760030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:49.074413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.071010    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:49.106033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.106001    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:49.296879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.296838    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"]
Oct 30 07:16:49.297925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.297890    6738 cpu_manager.go:414] "containerMap: removing container" podUID="057c1400-5914-476d-846b-42dab7eb3613" containerName="payload-pre-processing"
Oct 30 07:16:49.297925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.297922    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057c1400-5914-476d-846b-42dab7eb3613" containerName="payload-pre-processing"
Oct 30 07:16:49.298115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.297950    6738 cpu_manager.go:414] "containerMap: removing container" podUID="730c772e-adf0-41d3-88d3-2e428ce56e6c" containerName="payload-processing"
Oct 30 07:16:49.298115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.297958    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730c772e-adf0-41d3-88d3-2e428ce56e6c" containerName="payload-processing"
Oct 30 07:16:49.298192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.298133    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="057c1400-5914-476d-846b-42dab7eb3613" containerName="payload-pre-processing"
Oct 30 07:16:49.298192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.298154    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="730c772e-adf0-41d3-88d3-2e428ce56e6c" containerName="payload-processing"
Oct 30 07:16:49.301695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.301675    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.318836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.318810    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:16:49.329031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.328959    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"]
Oct 30 07:16:49.342233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.342204    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:16:49.342472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.342276    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:16:49.491029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.490994    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:16:49.653374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.528056    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.653807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.653766    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffzh\" (UniqueName: \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.654190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.654118    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.654733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.565775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:49.764317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.759684    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.764317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.759878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.764317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.760027    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffzh\" (UniqueName: \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.811384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.780195    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.817173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.817136    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffzh\" (UniqueName: \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.832128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.832093    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:49.833694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.833667    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:49.920738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:49.920636    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:50.397009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:50.396980    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"]
Oct 30 07:16:50.399605 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:50.399576    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843afc2f_4512_4b0c_bffe_aafdae44bbfa.slice/crio-8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078 WatchSource:0}: Error finding container 8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078: Status 404 returned error can't find the container with id 8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078
Oct 30 07:16:50.717896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:50.717799    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9" event={"ID":"843afc2f-4512-4b0c-bffe-aafdae44bbfa","Type":"ContainerStarted","Data":"8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078"}
Oct 30 07:16:51.588020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:51.587982    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:51.808527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:51.807687    6738 generic.go:356] "Generic (PLEG): container finished" podID="843afc2f-4512-4b0c-bffe-aafdae44bbfa" containerID="1010b908de74b53da101dc3e183fedde2257123b9f00e80263b77f0acb372207" exitCode=0
Oct 30 07:16:51.808527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:51.808004    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9" event={"ID":"843afc2f-4512-4b0c-bffe-aafdae44bbfa","Type":"ContainerDied","Data":"1010b908de74b53da101dc3e183fedde2257123b9f00e80263b77f0acb372207"}
Oct 30 07:16:53.031344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.031281    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:53.147061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.147024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\" (UniqueName: \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\") pod \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") "
Oct 30 07:16:53.147275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.147262    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\") pod \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") "
Oct 30 07:16:53.147468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.147449    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\") pod \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\" (UID: \"843afc2f-4512-4b0c-bffe-aafdae44bbfa\") "
Oct 30 07:16:53.147676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.147651    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca" pod "843afc2f-4512-4b0c-bffe-aafdae44bbfa" (UID: "843afc2f-4512-4b0c-bffe-aafdae44bbfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:53.148037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.148019    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:53.149265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.149240    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh" pod "843afc2f-4512-4b0c-bffe-aafdae44bbfa" (UID: "843afc2f-4512-4b0c-bffe-aafdae44bbfa"). InnerVolumeSpecName "kube-api-access-xffzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:53.149622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.149599    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca" pod "843afc2f-4512-4b0c-bffe-aafdae44bbfa" (UID: "843afc2f-4512-4b0c-bffe-aafdae44bbfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:16:53.151096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.151072    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:53.249455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.249419    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843afc2f-4512-4b0c-bffe-aafdae44bbfa-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:53.249455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.249448    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xffzh\" (UniqueName: \"kubernetes.io/projected/843afc2f-4512-4b0c-bffe-aafdae44bbfa-kube-api-access-xffzh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:53.383038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.383001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:53.945897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.945862    6738 generic.go:356] "Generic (PLEG): container finished" podID="7495b90f-0807-44c4-a023-7dfe4107408b" containerID="3daff7c6a342345b4cdd4aaa6f11f6a6d1be586be893f91a18102c3f19469b88" exitCode=0
Oct 30 07:16:53.946465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.946436    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerDied","Data":"3daff7c6a342345b4cdd4aaa6f11f6a6d1be586be893f91a18102c3f19469b88"}
Oct 30 07:16:53.946655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.946640    6738 scope.go:122] "RemoveContainer" containerID="a3bee4efcf6e7664063ba3be38ecb7a64d0a2fd231f2e93af252daaf868ed781"
Oct 30 07:16:53.959210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.959176    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:16:53.961046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.961019    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"
Oct 30 07:16:53.983086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.983056    6738 cpu_manager.go:414] "containerMap: removing container" podUID="843afc2f-4512-4b0c-bffe-aafdae44bbfa" containerName="revoke-keys"
Oct 30 07:16:53.983280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.983264    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843afc2f-4512-4b0c-bffe-aafdae44bbfa" containerName="revoke-keys"
Oct 30 07:16:53.983669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:53.983648    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="843afc2f-4512-4b0c-bffe-aafdae44bbfa" containerName="revoke-keys"
Oct 30 07:16:54.104379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.089879    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" podUID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" containerName="payload-processing" containerID="cri-o://b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a" gracePeriod=25
Oct 30 07:16:54.146814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.126311    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" podUID="c7120dc5-da24-4ddc-972e-bfb842a814c9" containerName="payload-pre-processing" containerID="cri-o://e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a" gracePeriod=25
Oct 30 07:16:54.146814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.142105    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9" event={"ID":"843afc2f-4512-4b0c-bffe-aafdae44bbfa","Type":"ContainerDied","Data":"8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078"}
Oct 30 07:16:54.146814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.142148    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df4eaf0e82a679fcc2fdfbecad894aea33b550b9c24c3921f077370e8e7c078"
Oct 30 07:16:54.146814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.142167    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:16:54.146814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.142348    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.157798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.157758    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:16:54.164606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.164546    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.165803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.165781    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-092c5c\""
Oct 30 07:16:54.211556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.211520    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-092c5c-dockercfg-wg2zt\""
Oct 30 07:16:54.224035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.223991    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.224191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.224171    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2j8t\" (UniqueName: \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.229596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.229519    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:16:54.325684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.325644    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.325834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.325771    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.325834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.325809    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2j8t\" (UniqueName: \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.325953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.325925    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.325960    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nffh\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326004    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326057    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326121    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326214    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326260    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.326379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.326351    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.327304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.327229    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.354209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.354129    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2j8t\" (UniqueName: \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\") pod \"payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.412763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.409512    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:16:54.425929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.425899    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:16:54.426090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.425887    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.427380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427213    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nffh\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427254    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427601    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427688    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427761    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427870    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.427996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.427947    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.428074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.428055    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.429209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.428862    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.429446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.429424    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.429683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.429666    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.429918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.429900    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.431086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.431062    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.432101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.431942    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.434052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.434026    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.466395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.466348    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nffh\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.500328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.467623    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.529757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.529571    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddbp\" (UniqueName: \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.529757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.529660    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.531126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.530651    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:16:54.621786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.621752    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.630819    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.630918    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.630996    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.631102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.631138    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.631178    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.631221    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.631593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.631393    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:54.632469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.632186    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.632594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.632487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:54.633114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.632681    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\") pod \"7495b90f-0807-44c4-a023-7dfe4107408b\" (UID: \"7495b90f-0807-44c4-a023-7dfe4107408b\") "
Oct 30 07:16:54.634254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.633556    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddbp\" (UniqueName: \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.634254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.633649    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.634254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.633895    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.634254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.633914    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.634254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.634068    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:54.635069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635001    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:54.635158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635077    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:54.635158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635126    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:54.635340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635320    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.635536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635518    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:54.635600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.635585    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:16:54.637714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.637694    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "kube-api-access-f4kvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:54.637941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.637921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy" pod "7495b90f-0807-44c4-a023-7dfe4107408b" (UID: "7495b90f-0807-44c4-a023-7dfe4107408b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:54.640154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.640133    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:54.653466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.653436    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:54.660481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.660443    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddbp\" (UniqueName: \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\") pod \"payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.734758    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\") pod \"c7120dc5-da24-4ddc-972e-bfb842a814c9\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") "
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.734993    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\" (UniqueName: \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\") pod \"c7120dc5-da24-4ddc-972e-bfb842a814c9\" (UID: \"c7120dc5-da24-4ddc-972e-bfb842a814c9\") "
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.735137    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\") pod \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") "
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.735402    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\" (UniqueName: \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\") pod \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\" (UID: \"77ad595c-4adc-4bee-b0c7-819ade9c61f7\") "
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.737281    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7495b90f-0807-44c4-a023-7dfe4107408b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.737680    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7495b90f-0807-44c4-a023-7dfe4107408b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.738831    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.738858    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.738875    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kvv\" (UniqueName: \"kubernetes.io/projected/7495b90f-0807-44c4-a023-7dfe4107408b-kube-api-access-f4kvv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.738891    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.738906    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7495b90f-0807-44c4-a023-7dfe4107408b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.739461    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume" pod "77ad595c-4adc-4bee-b0c7-819ade9c61f7" (UID: "77ad595c-4adc-4bee-b0c7-819ade9c61f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.739656    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume" pod "c7120dc5-da24-4ddc-972e-bfb842a814c9" (UID: "c7120dc5-da24-4ddc-972e-bfb842a814c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:54.742358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.740027    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77" pod "77ad595c-4adc-4bee-b0c7-819ade9c61f7" (UID: "77ad595c-4adc-4bee-b0c7-819ade9c61f7"). InnerVolumeSpecName "kube-api-access-5fp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:54.751382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.751331    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj" pod "c7120dc5-da24-4ddc-972e-bfb842a814c9" (UID: "c7120dc5-da24-4ddc-972e-bfb842a814c9"). InnerVolumeSpecName "kube-api-access-r77jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:54.829639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.829605    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:54.841168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.841132    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ad595c-4adc-4bee-b0c7-819ade9c61f7-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.841168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.841163    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fp77\" (UniqueName: \"kubernetes.io/projected/77ad595c-4adc-4bee-b0c7-819ade9c61f7-kube-api-access-5fp77\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.841168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.841175    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7120dc5-da24-4ddc-972e-bfb842a814c9-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.841516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.841184    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77jj\" (UniqueName: \"kubernetes.io/projected/c7120dc5-da24-4ddc-972e-bfb842a814c9-kube-api-access-r77jj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:16:54.976202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.976166    6738 generic.go:356] "Generic (PLEG): container finished" podID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" containerID="b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a" exitCode=0
Oct 30 07:16:54.976397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.976302    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"
Oct 30 07:16:54.976397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.976347    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" event={"ID":"77ad595c-4adc-4bee-b0c7-819ade9c61f7","Type":"ContainerDied","Data":"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"}
Oct 30 07:16:54.976532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.976416    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms" event={"ID":"77ad595c-4adc-4bee-b0c7-819ade9c61f7","Type":"ContainerDied","Data":"e99170be9c0d009595a7734ea967388e121b979e5f5de7d999449ce067c5d044"}
Oct 30 07:16:54.976532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.976448    6738 scope.go:122] "RemoveContainer" containerID="b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"
Oct 30 07:16:54.985470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.981783    6738 generic.go:356] "Generic (PLEG): container finished" podID="c7120dc5-da24-4ddc-972e-bfb842a814c9" containerID="e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a" exitCode=0
Oct 30 07:16:54.985470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.982038    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" event={"ID":"c7120dc5-da24-4ddc-972e-bfb842a814c9","Type":"ContainerDied","Data":"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"}
Oct 30 07:16:54.985470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.982086    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2" event={"ID":"c7120dc5-da24-4ddc-972e-bfb842a814c9","Type":"ContainerDied","Data":"53c0d40c4f8247e4c60c7c977a9fa6c35f67ff4693a2adafb77233f10ddc7fe5"}
Oct 30 07:16:54.985470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.982170    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"
Oct 30 07:16:54.990092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.988274    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n" event={"ID":"7495b90f-0807-44c4-a023-7dfe4107408b","Type":"ContainerDied","Data":"c0e74d16fa995f8c6282150b28b1efc9f383f7c0e102ecaf0986b87c06d2b002"}
Oct 30 07:16:54.990092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.988335    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"
Oct 30 07:16:54.998603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.998501    6738 scope.go:122] "RemoveContainer" containerID="b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"
Oct 30 07:16:54.999161 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:54.999026    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a\": container with ID starting with b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a not found: ID does not exist" containerID="b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"
Oct 30 07:16:54.999161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.999061    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a"} err="failed to get container status \"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a\": rpc error: code = NotFound desc = could not find container \"b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a\": container with ID starting with b1d937cd5d36c54439a10b628434a9b9479bf23d489731f19edc4c72bd91d83a not found: ID does not exist"
Oct 30 07:16:54.999161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:54.999085    6738 scope.go:122] "RemoveContainer" containerID="e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"
Oct 30 07:16:55.009508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.009443    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:16:55.028661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.028604    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:55.034830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.034801    6738 scope.go:122] "RemoveContainer" containerID="e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"
Oct 30 07:16:55.035378 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:16:55.035226    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a\": container with ID starting with e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a not found: ID does not exist" containerID="e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"
Oct 30 07:16:55.035378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.035264    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a"} err="failed to get container status \"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a\": rpc error: code = NotFound desc = could not find container \"e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a\": container with ID starting with e6869dd11961dfa6e61fb7412b5b6e552150005f2d237abad0af59043333cb8a not found: ID does not exist"
Oct 30 07:16:55.035692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.035288    6738 scope.go:122] "RemoveContainer" containerID="3daff7c6a342345b4cdd4aaa6f11f6a6d1be586be893f91a18102c3f19469b88"
Oct 30 07:16:55.082723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.082693    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:16:55.143318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.142087    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:55.145550 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:16:55.145510    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4677d08e_758f_435f_a3cb_9e61305b1de2.slice/crio-f8e8d72c1b332eeaa1609b949980ab782e3ff41f07a09ec0edefe74f9537cd7c WatchSource:0}: Error finding container f8e8d72c1b332eeaa1609b949980ab782e3ff41f07a09ec0edefe74f9537cd7c: Status 404 returned error can't find the container with id f8e8d72c1b332eeaa1609b949980ab782e3ff41f07a09ec0edefe74f9537cd7c
Oct 30 07:16:55.180019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.179955    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:55.180672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.180384    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:55.180902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.180874    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:16:55.184653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.184624    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-5f5f8fc9c6-qxdms"]
Oct 30 07:16:55.231231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.231147    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:55.239749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.239433    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-8649f69f9f-q7sv2"]
Oct 30 07:16:55.248757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.247565    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:16:55.264546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.264402    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:16:55.264546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.264476    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c0e3de-openshift-default-5497cd949-88n2n"]
Oct 30 07:16:55.695146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.695107    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" path="/var/lib/kubelet/pods/7495b90f-0807-44c4-a023-7dfe4107408b/volumes"
Oct 30 07:16:55.695764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.695747    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" path="/var/lib/kubelet/pods/77ad595c-4adc-4bee-b0c7-819ade9c61f7/volumes"
Oct 30 07:16:55.696264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:55.696250    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7120dc5-da24-4ddc-972e-bfb842a814c9" path="/var/lib/kubelet/pods/c7120dc5-da24-4ddc-972e-bfb842a814c9/volumes"
Oct 30 07:16:56.012352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.012215    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" event={"ID":"37d1bdca-7123-428e-9be1-61cd72e855a2","Type":"ContainerStarted","Data":"e8eb16adffbf8217de59f61afacba3048cda81c3324b2f5d84786f39bd9b384e"}
Oct 30 07:16:56.012352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.012339    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" event={"ID":"37d1bdca-7123-428e-9be1-61cd72e855a2","Type":"ContainerStarted","Data":"f2fe11e98a6bb8b43b94efffcab7fd0a5b56b5c924f1f55758bcc1146bb65c79"}
Oct 30 07:16:56.012641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.012605    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:16:56.020528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.020491    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" event={"ID":"4677d08e-758f-435f-a3cb-9e61305b1de2","Type":"ContainerStarted","Data":"219c3132dc685c2c633532235d0ee76f949299ae8e99027a68a1a1740fe65709"}
Oct 30 07:16:56.020709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.020547    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" event={"ID":"4677d08e-758f-435f-a3cb-9e61305b1de2","Type":"ContainerStarted","Data":"f8e8d72c1b332eeaa1609b949980ab782e3ff41f07a09ec0edefe74f9537cd7c"}
Oct 30 07:16:56.022510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.022474    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" event={"ID":"b823a536-42e3-41fc-bf1c-ee35d171efdd","Type":"ContainerStarted","Data":"f3fe6d3d7006fa2401a6bc3b09f0446ec1d83be762a8828c4ccc621d42f981df"}
Oct 30 07:16:56.022679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.022529    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" event={"ID":"b823a536-42e3-41fc-bf1c-ee35d171efdd","Type":"ContainerStarted","Data":"59eabc0604f923b38e3bd7851735b4ec46868f054ff5aad57ed4d179ba2b4bb3"}
Oct 30 07:16:56.022679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.022559    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:16:56.118829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.118761    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" podStartSLOduration=3.118743604 podStartE2EDuration="3.118743604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:56.068572544 +0000 UTC m=+1176.501336465" watchObservedRunningTime="2026-10-30 07:16:56.118743604 +0000 UTC m=+1176.551507494"
Oct 30 07:16:56.181371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.179515    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podStartSLOduration=2.179495388 podStartE2EDuration="2.179495388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:56.169975973 +0000 UTC m=+1176.602739868" watchObservedRunningTime="2026-10-30 07:16:56.179495388 +0000 UTC m=+1176.612259283"
Oct 30 07:16:56.181371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.179972    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" podStartSLOduration=2.179957979 podStartE2EDuration="2.179957979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:56.123749879 +0000 UTC m=+1176.556513773" watchObservedRunningTime="2026-10-30 07:16:56.179957979 +0000 UTC m=+1176.612721874"
Oct 30 07:16:56.635082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.635044    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:16:56.636515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.636488    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:16:56.636609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.636540    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:16:56.913027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:56.912933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:57.624484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.624440    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:16:57.624897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.624517    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:16:57.636015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.635973    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:16:57.636176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.636037    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:16:57.782020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.781914    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:57.902402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:57.899602    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:58.032938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.032875    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"]
Oct 30 07:16:58.048260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.045195    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"]
Oct 30 07:16:58.050219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.050187    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c1fda0a-7zcmd"]
Oct 30 07:16:58.054698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.054669    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f86232a-n8mmf"]
Oct 30 07:16:58.636146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.636106    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:16:58.636657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:58.636173    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:16:59.636096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.636058    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:16:59.636276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.636129    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:16:59.700211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.700165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:16:59.702007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.701980    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3880509e-eef0-40ad-b522-bb1feaac449c" path="/var/lib/kubelet/pods/3880509e-eef0-40ad-b522-bb1feaac449c/volumes"
Oct 30 07:16:59.702535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.702520    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a5c09b-9f1c-4696-a281-d09163484bd5" path="/var/lib/kubelet/pods/e7a5c09b-9f1c-4696-a281-d09163484bd5/volumes"
Oct 30 07:16:59.935823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:16:59.933470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:00.098219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:00.094000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:00.639810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:00.639774    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:17:00.640569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:00.640536    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:17:01.203079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.203034    6738 generic.go:356] "Generic (PLEG): container finished" podID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerID="f372e1c8d8d4067add08641451a75dc19973043aa94bbe2ae500d67a4226393e" exitCode=0
Oct 30 07:17:01.203316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.203160    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerDied","Data":"f372e1c8d8d4067add08641451a75dc19973043aa94bbe2ae500d67a4226393e"}
Oct 30 07:17:01.203843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.203818    6738 scope.go:122] "RemoveContainer" containerID="f372e1c8d8d4067add08641451a75dc19973043aa94bbe2ae500d67a4226393e"
Oct 30 07:17:01.205941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.205905    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:01.206054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.205995    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:01.206116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.206062    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:01.635449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.635411    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 07:17:01.635650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.635462    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 07:17:01.929356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:01.929248    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:02.216217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.216119    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerStarted","Data":"fcc41e4ff22ce58e4ccdb3ff87fb3a85684f5670dce1b8ed19421e37aeac2d7e"}
Oct 30 07:17:02.451327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.437381    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:17:02.451327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.438472    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:02.451327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.438539    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:02.707316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.702667    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:17:02.721551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.721317    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:17:02.742328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:02.726576    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:17:03.437316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:03.436465    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:03.437316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:03.436542    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:04.004361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.004314    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:04.123822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.118494    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:04.435843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.435808    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:04.436028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.435880    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:04.439654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.439625    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:17:04.440000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.439868    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" containerID="cri-o://0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b" gracePeriod=30
Oct 30 07:17:04.526363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.526320    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:04.790741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.790615    6738 patch_prober.go:29] interesting pod/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:17:04.791109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:04.791079    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:17:05.012471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:05.012432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:05.152756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:05.152717    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:05.436154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:05.436067    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:05.436351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:05.436149    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:06.321320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:06.319274    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:06.438726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:06.438686    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:06.439031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:06.438999    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:06.460328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:06.458904    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:06.669258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:06.669220    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:07.047768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:07.047522    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:17:07.047946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:07.047914    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:17:07.435998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:07.435957    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:07.436476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:07.436030    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:07.544829 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:07.544785    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:17:08.230524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.230484    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"]
Oct 30 07:17:08.231278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231255    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c7120dc5-da24-4ddc-972e-bfb842a814c9" containerName="payload-pre-processing"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231280    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7120dc5-da24-4ddc-972e-bfb842a814c9" containerName="payload-pre-processing"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231343    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231351    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231383    6738 cpu_manager.go:414] "containerMap: removing container" podUID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" containerName="payload-processing"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231391    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" containerName="payload-processing"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231418    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:08.231446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.231425    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:08.232213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.232176    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:08.232355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.232221    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7120dc5-da24-4ddc-972e-bfb842a814c9" containerName="payload-pre-processing"
Oct 30 07:17:08.232355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.232247    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ad595c-4adc-4bee-b0c7-819ade9c61f7" containerName="payload-processing"
Oct 30 07:17:08.236025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.236005    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.257736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.257702    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"]
Oct 30 07:17:08.263306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.263260    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:17:08.263548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.263269    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:17:08.292321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.268073    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb2r\" (UniqueName: \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.292722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.292685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.292935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.292916    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.329635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.329599    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:17:08.329895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.329877    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:17:08.400910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.400877    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb2r\" (UniqueName: \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.401241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.401217    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.401470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.401436    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.402274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.402234    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.403847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.403823    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.424968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.424936    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb2r\" (UniqueName: \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\") pod \"maas-api-revoke-sub-8a693bf6-vsnrf\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.436247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.436210    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 07:17:08.436778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.436275    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 07:17:08.554436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.554342    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:08.732280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.732170    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:08.958405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:08.958379    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"]
Oct 30 07:17:08.960837 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:08.960603    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b72dc59_83c0_49e7_a726_cd931c7ef91f.slice/crio-89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853 WatchSource:0}: Error finding container 89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853: Status 404 returned error can't find the container with id 89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853
Oct 30 07:17:09.398103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.398064    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf" event={"ID":"6b72dc59-83c0-49e7-a726-cd931c7ef91f","Type":"ContainerStarted","Data":"89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853"}
Oct 30 07:17:09.439796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.439759    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:17:09.440203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.440140    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:17:09.440960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.440938    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:17:09.839012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.838989    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:17:09.941832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.941791    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.941890    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.941950    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942203    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942419    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942543    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942632    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942756    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942816    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\") pod \"2d42ae1b-497f-41d6-9129-882c6e4020ff\" (UID: \"2d42ae1b-497f-41d6-9129-882c6e4020ff\") "
Oct 30 07:17:09.942929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.942895    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:09.943234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.943209    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:09.943623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.943561    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:09.943727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.943685    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d42ae1b-497f-41d6-9129-882c6e4020ff-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:09.943920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.943801    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:09.943920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.943892    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:09.944366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.944344    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:09.944867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.944842    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:09.945056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.945024    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:09.945752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.945726    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "kube-api-access-sfn9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:09.945881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:09.945865    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo" pod "2d42ae1b-497f-41d6-9129-882c6e4020ff" (UID: "2d42ae1b-497f-41d6-9129-882c6e4020ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:10.045602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045561    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045595    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045604    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045616    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045627    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn9d\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-kube-api-access-sfn9d\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045635    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.045889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.045662    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d42ae1b-497f-41d6-9129-882c6e4020ff-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:10.229211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.229169    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:10.441548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.431491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:10.484889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.484855    6738 generic.go:356] "Generic (PLEG): container finished" podID="6b72dc59-83c0-49e7-a726-cd931c7ef91f" containerID="55328ea7571eae6cc0e398eaeece63d4993bfddff8e566f8721d92761a02e34d" exitCode=0
Oct 30 07:17:10.485133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.485043    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf" event={"ID":"6b72dc59-83c0-49e7-a726-cd931c7ef91f","Type":"ContainerDied","Data":"55328ea7571eae6cc0e398eaeece63d4993bfddff8e566f8721d92761a02e34d"}
Oct 30 07:17:10.488673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.488603    6738 generic.go:356] "Generic (PLEG): container finished" podID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerID="0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b" exitCode=0
Oct 30 07:17:10.489984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.489586    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"
Oct 30 07:17:10.504912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.502538    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerDied","Data":"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"}
Oct 30 07:17:10.504912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.502615    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr" event={"ID":"2d42ae1b-497f-41d6-9129-882c6e4020ff","Type":"ContainerDied","Data":"0b1601706aa50ed4d73e5ede34a6a79ee731e1a8f25d92c6dbc5b827abcca2d8"}
Oct 30 07:17:10.504912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.502646    6738 scope.go:122] "RemoveContainer" containerID="0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"
Oct 30 07:17:10.706394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.680250    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:17:10.769339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.747680    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a6a14e07-openshift-default-584d96d889-6xwpr"]
Oct 30 07:17:10.769339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.754482    6738 scope.go:122] "RemoveContainer" containerID="de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.778265    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784362    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784397    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784421    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784429    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784731    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784747    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" containerName="istio-proxy"
Oct 30 07:17:10.813508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.784757    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7495b90f-0807-44c4-a023-7dfe4107408b" containerName="istio-proxy"
Oct 30 07:17:10.931398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.916535    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:17:10.931398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:10.916683    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.005452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.005416    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba67d8-dockercfg-bmwnk\""
Oct 30 07:17:11.006209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.006187    6738 scope.go:122] "RemoveContainer" containerID="0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"
Oct 30 07:17:11.006897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.006875    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba67d8\""
Oct 30 07:17:11.010690 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:11.010641    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b\": container with ID starting with 0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b not found: ID does not exist" containerID="0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"
Oct 30 07:17:11.010817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.010700    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b"} err="failed to get container status \"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b\": rpc error: code = NotFound desc = could not find container \"0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b\": container with ID starting with 0750fb929296eed203581112a050d97fe63b3699e272feec19898ecb3d0a1a3b not found: ID does not exist"
Oct 30 07:17:11.010817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.010730    6738 scope.go:122] "RemoveContainer" containerID="de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"
Oct 30 07:17:11.011400 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:11.011346    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d\": container with ID starting with de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d not found: ID does not exist" containerID="de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"
Oct 30 07:17:11.011400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.011385    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d"} err="failed to get container status \"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d\": rpc error: code = NotFound desc = could not find container \"de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d\": container with ID starting with de738fc119b8a9cc363272c910212fc206b28eff72243e124c3f74f137f9928d not found: ID does not exist"
Oct 30 07:17:11.043092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.043040    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.043268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.043204    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6td\" (UniqueName: \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.055635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.055589    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:17:11.067845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.067812    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.068045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.067987    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:17:11.148590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.148550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.149042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.149016    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6td\" (UniqueName: \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.149665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.149632    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.173584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.173551    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6td\" (UniqueName: \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\") pod \"payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.251370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.251238    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rc7n\" (UniqueName: \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.251563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.251539    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.353334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.353275    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rc7n\" (UniqueName: \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.353588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.353571    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.354308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.354264    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.377519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.377488    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rc7n\" (UniqueName: \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\") pod \"payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.432572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.432534    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:11.437085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.437062    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:11.521848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.521817    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:11.722217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.720863    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d42ae1b-497f-41d6-9129-882c6e4020ff" path="/var/lib/kubelet/pods/2d42ae1b-497f-41d6-9129-882c6e4020ff/volumes"
Oct 30 07:17:11.876824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.876788    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:11.912556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.912527    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\") pod \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") "
Oct 30 07:17:11.912556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.912666    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\" (UniqueName: \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\") pod \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") "
Oct 30 07:17:11.912926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.912890    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\") pod \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\" (UID: \"6b72dc59-83c0-49e7-a726-cd931c7ef91f\") "
Oct 30 07:17:11.913452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.913419    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca" pod "6b72dc59-83c0-49e7-a726-cd931c7ef91f" (UID: "6b72dc59-83c0-49e7-a726-cd931c7ef91f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:11.913614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.913544    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:11.915329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.915283    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca" pod "6b72dc59-83c0-49e7-a726-cd931c7ef91f" (UID: "6b72dc59-83c0-49e7-a726-cd931c7ef91f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:17:11.915444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.915416    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r" pod "6b72dc59-83c0-49e7-a726-cd931c7ef91f" (UID: "6b72dc59-83c0-49e7-a726-cd931c7ef91f"). InnerVolumeSpecName "kube-api-access-whb2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:11.959178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.959145    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:17:11.970658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:11.970143    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:17:11.979781 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:11.979747    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac45e3e_179c_4aa8_8a71_5bf626280bbc.slice/crio-6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f WatchSource:0}: Error finding container 6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f: Status 404 returned error can't find the container with id 6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f
Oct 30 07:17:12.014983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.014949    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whb2r\" (UniqueName: \"kubernetes.io/projected/6b72dc59-83c0-49e7-a726-cd931c7ef91f-kube-api-access-whb2r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:12.014983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.014978    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b72dc59-83c0-49e7-a726-cd931c7ef91f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:12.512998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.512919    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" event={"ID":"0ac45e3e-179c-4aa8-8a71-5bf626280bbc","Type":"ContainerStarted","Data":"ed0eb6445a3ea5284828a1a128374c9795fe4a02fb3c396d502521af1da31eb2"}
Oct 30 07:17:12.512998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.512998    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:12.513271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.513024    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" event={"ID":"0ac45e3e-179c-4aa8-8a71-5bf626280bbc","Type":"ContainerStarted","Data":"6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f"}
Oct 30 07:17:12.515103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.515068    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" event={"ID":"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e","Type":"ContainerStarted","Data":"e9751e7568da2f3b45250a145c5a39339593fe56bc7545d0fd21abf41616b183"}
Oct 30 07:17:12.515250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.515130    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:12.515250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.515158    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" event={"ID":"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e","Type":"ContainerStarted","Data":"1f9068389771173b8d0c633382f06a10185e56f9a9b6b291da7783defffef0e5"}
Oct 30 07:17:12.516854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.516832    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"
Oct 30 07:17:12.516854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.516841    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf" event={"ID":"6b72dc59-83c0-49e7-a726-cd931c7ef91f","Type":"ContainerDied","Data":"89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853"}
Oct 30 07:17:12.517015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.516889    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89f16a8153f213bbbc3574b39de5a8f07e73760db5298fbf03feb22827d19853"
Oct 30 07:17:12.557120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.557058    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" podStartSLOduration=2.557040549 podStartE2EDuration="2.557040549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:12.554099894 +0000 UTC m=+1192.986863788" watchObservedRunningTime="2026-10-30 07:17:12.557040549 +0000 UTC m=+1192.989804443"
Oct 30 07:17:12.599915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:12.599853    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" podStartSLOduration=2.599839071 podStartE2EDuration="2.599839071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:12.595796047 +0000 UTC m=+1193.028559968" watchObservedRunningTime="2026-10-30 07:17:12.599839071 +0000 UTC m=+1193.032602965"
Oct 30 07:17:13.511517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:13.511484    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:15.248919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.248881    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:15.546685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.546584    6738 generic.go:356] "Generic (PLEG): container finished" podID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerID="e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848" exitCode=0
Oct 30 07:17:15.546685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.546665    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerDied","Data":"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"}
Oct 30 07:17:15.547281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.547265    6738 scope.go:122] "RemoveContainer" containerID="e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"
Oct 30 07:17:15.549012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.548974    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:15.549147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.549069    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:15.549147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:15.549119    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:16.557276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:16.557241    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerStarted","Data":"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"}
Oct 30 07:17:16.834107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:16.834018    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:17:16.835137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:16.835105    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:16.835258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:16.835157    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:17.034395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:17.034353    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr_b823a536-42e3-41fc-bf1c-ee35d171efdd/payload-processing/0.log"
Oct 30 07:17:17.116717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:17.116626    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:17.834208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:17.834170    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:17.834717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:17.834238    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:18.759182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.759151    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:17:18.801124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.801091    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:17:18.806379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.806338    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:18.833652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.833620    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:18.833821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.833675    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:18.949025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:18.948984    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:19.035024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.034928    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"]
Oct 30 07:17:19.040239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.040195    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd706324-k7bng"]
Oct 30 07:17:19.089090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.085934    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:19.242333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.233061    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:19.866317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.865513    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:19.866317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:19.865582    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:20.020264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.020217    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4816a68-f98c-46ab-9a66-3a803f868e82" path="/var/lib/kubelet/pods/f4816a68-f98c-46ab-9a66-3a803f868e82/volumes"
Oct 30 07:17:20.327903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.327862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:20.404940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.404902    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:17:20.414241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.414206    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6b72dc59-83c0-49e7-a726-cd931c7ef91f" containerName="revoke-keys"
Oct 30 07:17:20.414505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.414485    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b72dc59-83c0-49e7-a726-cd931c7ef91f" containerName="revoke-keys"
Oct 30 07:17:20.414886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.414866    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b72dc59-83c0-49e7-a726-cd931c7ef91f" containerName="revoke-keys"
Oct 30 07:17:20.434057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.434029    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:20.454382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.454286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:20.481413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.480697    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a273fd\"/\"kube-root-ca.crt\""
Oct 30 07:17:20.481413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.480788    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a273fd\"/\"openshift-service-ca.crt\""
Oct 30 07:17:20.481413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.480889    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a273fd\"/\"default-dockercfg-s75cl\""
Oct 30 07:17:20.481413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.480956    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a273fd\"/\"ipp-route-a273fd-kserve-self-signed-certs\""
Oct 30 07:17:20.991033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.990989    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfd8n\" (UniqueName: \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:20.991434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.991397    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:20.991640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.991620    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:20.994626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.994597    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.004318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.994793    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.005038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.005014    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.018519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:20.992367    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:17:21.037311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.037262    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:21.037755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.037340    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:21.042036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.041998    6738 scope.go:122] "RemoveContainer" containerID="6aa2895b02a1e73ee3fb631e3a43b2f2be5e98470fb6ecb0992a1ec9be413167"
Oct 30 07:17:21.104339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.093704    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:17:21.124527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.124500    6738 scope.go:122] "RemoveContainer" containerID="e223e85762137cbdc21650cd75906095ac86c3f84c1deccefa21010bdf04d5e1"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136450    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136534    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136833    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfd8n\" (UniqueName: \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136894    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136938    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.136977    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.139258    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.139518    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.140140    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"]
Oct 30 07:17:21.141415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.140424    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.153315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.149599    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7738eaca-zb74b"]
Oct 30 07:17:21.195319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.195157    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.228318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.203547    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfd8n\" (UniqueName: \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.255367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.255281    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\") pod \"ipp-route-a273fd-kserve-578b47dbb9-wxxxk\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") " pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.406322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.383516    6738 scope.go:122] "RemoveContainer" containerID="5ec1db34cb937b03cf8f532916af362e6d40557aca0ea7a7dbc20a6620a33fad"
Oct 30 07:17:21.440321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.429209    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:21.533330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.529489    6738 scope.go:122] "RemoveContainer" containerID="d057ba9198accdb37ad779c8975f0ec789695ed8fa02b0df8f79f72b2ef2c0a1"
Oct 30 07:17:21.634952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.634924    6738 scope.go:122] "RemoveContainer" containerID="9a5dcae311b4e2a9fbea20237bb707ac614c9cff13507f4b7517cff3219afc0d"
Oct 30 07:17:21.773145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.772098    6738 scope.go:122] "RemoveContainer" containerID="3998a28c7696f1a0c9b4ae5d5e0773c2bea7158cd351d838cf6372313c9124a1"
Oct 30 07:17:21.825374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.812746    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4c0d6c-f15a-4651-be89-cd1235d5943a" path="/var/lib/kubelet/pods/da4c0d6c-f15a-4651-be89-cd1235d5943a/volumes"
Oct 30 07:17:21.833940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.833909    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:21.834232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.834184    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:21.903713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.903678    6738 scope.go:122] "RemoveContainer" containerID="ff4f301fd9581be5a819012ae4d81745e290872dfbc0fecc12bdee47582d3595"
Oct 30 07:17:22.024320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:21.998113    6738 scope.go:122] "RemoveContainer" containerID="7c056e954a7e2fc63a7ede145c60d88367c78765cc1cb57e23a92d3e5fcf304f"
Oct 30 07:17:22.502466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.502356    6738 scope.go:122] "RemoveContainer" containerID="001ddbc14c52fcaf8eedf2b37ebb78fcf072da11198605fa6048265414188099"
Oct 30 07:17:22.527638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.526421    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:17:22.688325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.669608    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:17:22.759824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.758394    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:17:22.865426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.855386    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:17:22.892364 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:22.870620    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e49d9ec_ceae_4a6d_8a63_9690b5c429f1.slice/crio-a6ed400ee626c306663b55238baece850faa2b1780847d641c8d26c66f7e380c WatchSource:0}: Error finding container a6ed400ee626c306663b55238baece850faa2b1780847d641c8d26c66f7e380c: Status 404 returned error can't find the container with id a6ed400ee626c306663b55238baece850faa2b1780847d641c8d26c66f7e380c
Oct 30 07:17:22.892364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.870701    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:17:22.926337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.900570    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 07:17:22.926337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:22.900633    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 07:17:23.152945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.152393    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:17:23.160699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.160648    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:17:23.487890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.487850    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" event={"ID":"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1","Type":"ContainerStarted","Data":"569216576cd988df6a610886bc079c7b39e83c80661e2609ff73a8b1b584c0b7"}
Oct 30 07:17:23.488071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.487898    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" event={"ID":"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1","Type":"ContainerStarted","Data":"a6ed400ee626c306663b55238baece850faa2b1780847d641c8d26c66f7e380c"}
Oct 30 07:17:23.528273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.528236    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:17:23.528784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.528339    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:17:23.532922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.532872    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" podStartSLOduration=3.532857693 podStartE2EDuration="3.532857693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:23.531191098 +0000 UTC m=+1203.963954991" watchObservedRunningTime="2026-10-30 07:17:23.532857693 +0000 UTC m=+1203.965621586"
Oct 30 07:17:23.760518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.760425    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" podUID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" containerName="payload-processing" containerID="cri-o://1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418" gracePeriod=25
Oct 30 07:17:23.766220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.766192    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:23.801634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.801589    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" podUID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" containerName="payload-pre-processing" containerID="cri-o://aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9" gracePeriod=25
Oct 30 07:17:23.921977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.921930    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:17:23.958117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:23.949735    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:17:24.205748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.205720    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:17:24.218165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.218144    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:17:24.290855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.290816    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\" (UniqueName: \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\") pod \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") "
Oct 30 07:17:24.291403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.290963    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\") pod \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\" (UID: \"b8f855cb-7fd2-44f8-9ebd-a312aef1814d\") "
Oct 30 07:17:24.291684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.291447    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\" (UniqueName: \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\") pod \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") "
Oct 30 07:17:24.291684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.291570    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\") pod \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\" (UID: \"ae65b93c-f27a-4fd4-a142-8219f7a59a76\") "
Oct 30 07:17:24.291684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.291613    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume" pod "b8f855cb-7fd2-44f8-9ebd-a312aef1814d" (UID: "b8f855cb-7fd2-44f8-9ebd-a312aef1814d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:24.292727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.292688    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume" pod "ae65b93c-f27a-4fd4-a142-8219f7a59a76" (UID: "ae65b93c-f27a-4fd4-a142-8219f7a59a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:24.296147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.296116    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6" pod "ae65b93c-f27a-4fd4-a142-8219f7a59a76" (UID: "ae65b93c-f27a-4fd4-a142-8219f7a59a76"). InnerVolumeSpecName "kube-api-access-wcrv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:24.296316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.296186    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6" pod "b8f855cb-7fd2-44f8-9ebd-a312aef1814d" (UID: "b8f855cb-7fd2-44f8-9ebd-a312aef1814d"). InnerVolumeSpecName "kube-api-access-v6vs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:24.393986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.393946    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vs6\" (UniqueName: \"kubernetes.io/projected/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-kube-api-access-v6vs6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:24.393986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.393979    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f855cb-7fd2-44f8-9ebd-a312aef1814d-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:24.393986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.393988    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrv6\" (UniqueName: \"kubernetes.io/projected/ae65b93c-f27a-4fd4-a142-8219f7a59a76-kube-api-access-wcrv6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:24.394225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.393997    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae65b93c-f27a-4fd4-a142-8219f7a59a76-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:24.437052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.437019    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:24.496772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.496736    6738 generic.go:356] "Generic (PLEG): container finished" podID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" containerID="1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418" exitCode=0
Oct 30 07:17:24.496965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.496780    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"
Oct 30 07:17:24.496965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.496827    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" event={"ID":"ae65b93c-f27a-4fd4-a142-8219f7a59a76","Type":"ContainerDied","Data":"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"}
Oct 30 07:17:24.496965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.496880    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg" event={"ID":"ae65b93c-f27a-4fd4-a142-8219f7a59a76","Type":"ContainerDied","Data":"b831c3596f0648ad9b4751b507c8284be7c353ff3173af6bcebb387974f7a0f8"}
Oct 30 07:17:24.496965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.496906    6738 scope.go:122] "RemoveContainer" containerID="1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"
Oct 30 07:17:24.498387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.498359    6738 generic.go:356] "Generic (PLEG): container finished" podID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" containerID="aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9" exitCode=0
Oct 30 07:17:24.498535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.498411    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"
Oct 30 07:17:24.498535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.498442    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" event={"ID":"b8f855cb-7fd2-44f8-9ebd-a312aef1814d","Type":"ContainerDied","Data":"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"}
Oct 30 07:17:24.498535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.498486    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz" event={"ID":"b8f855cb-7fd2-44f8-9ebd-a312aef1814d","Type":"ContainerDied","Data":"f9bc5844a0380d624d7d18cd915c6796c22cf1e43d2a199a0f302f3b775a3741"}
Oct 30 07:17:24.508713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.508574    6738 scope.go:122] "RemoveContainer" containerID="1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"
Oct 30 07:17:24.509088 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:24.509069    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418\": container with ID starting with 1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418 not found: ID does not exist" containerID="1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"
Oct 30 07:17:24.509166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.509098    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418"} err="failed to get container status \"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418\": rpc error: code = NotFound desc = could not find container \"1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418\": container with ID starting with 1e46c5aa9f62a9089e28d28b825355edd7782e35977a8e1e75bcb11c2f501418 not found: ID does not exist"
Oct 30 07:17:24.509166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.509116    6738 scope.go:122] "RemoveContainer" containerID="aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"
Oct 30 07:17:24.519109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.519090    6738 scope.go:122] "RemoveContainer" containerID="aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"
Oct 30 07:17:24.519461 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:24.519438    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9\": container with ID starting with aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9 not found: ID does not exist" containerID="aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"
Oct 30 07:17:24.519552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.519466    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9"} err="failed to get container status \"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9\": rpc error: code = NotFound desc = could not find container \"aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9\": container with ID starting with aa1007d9008f4af8026e0e16dacfe024f2e7e041ac0368a02a317ceb8d56aec9 not found: ID does not exist"
Oct 30 07:17:24.550386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.550346    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:17:24.554373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.554344    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-6d5d9bf754-cqqjg"]
Oct 30 07:17:24.589975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.589884    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:17:24.593929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:24.593902    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-5dd4ffd475-zqmmz"]
Oct 30 07:17:25.155405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:25.155373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:25.625261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:25.625228    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:25.716336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:25.714460    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" path="/var/lib/kubelet/pods/ae65b93c-f27a-4fd4-a142-8219f7a59a76/volumes"
Oct 30 07:17:25.716336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:25.715240    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" path="/var/lib/kubelet/pods/b8f855cb-7fd2-44f8-9ebd-a312aef1814d/volumes"
Oct 30 07:17:26.607496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:26.607451    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:27.033551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.033518    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:17:27.034228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034211    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" containerName="payload-processing"
Oct 30 07:17:27.034228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034226    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" containerName="payload-processing"
Oct 30 07:17:27.034437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034242    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" containerName="payload-pre-processing"
Oct 30 07:17:27.034437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034248    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" containerName="payload-pre-processing"
Oct 30 07:17:27.034437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034384    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae65b93c-f27a-4fd4-a142-8219f7a59a76" containerName="payload-processing"
Oct 30 07:17:27.034437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.034395    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f855cb-7fd2-44f8-9ebd-a312aef1814d" containerName="payload-pre-processing"
Oct 30 07:17:27.038284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.038263    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.067701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.067669    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c1f203-openshift-default-dockercfg-jf26b\""
Oct 30 07:17:27.074194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.074157    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:17:27.096971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.096939    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:27.187318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.185853    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.187318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.185923    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhrp\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.187318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.186001    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.187318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.186082    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.209345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.196209    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.209345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.196361    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.209345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.196605    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.209345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.196675    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.209345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.196716    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.289470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.289363    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:27.298485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.298444    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.304456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.304411    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.304726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.304705    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.304895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.304878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.305022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.305007    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhrp\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.305182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.305165    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.305455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.305431    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.305674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.305657    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.305839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.305825    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.318946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.318914    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.328863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.327252    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.329056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.327896    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.329169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.328522    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.329259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.328788    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.347170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.347133    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.355567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.355491    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhrp\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.370683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.370648    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.381316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.380994    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\") pod \"e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") " pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.656567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.656529    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:27.697896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.697859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:27.985909 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:27.985871    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7cefa8_8775_44f6_8b8d_3170c6b80303.slice/crio-0b91723346ce4cd8fa217924a3bec17caec49cd2966638542495c7f6fb77a5db WatchSource:0}: Error finding container 0b91723346ce4cd8fa217924a3bec17caec49cd2966638542495c7f6fb77a5db: Status 404 returned error can't find the container with id 0b91723346ce4cd8fa217924a3bec17caec49cd2966638542495c7f6fb77a5db
Oct 30 07:17:27.986603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.986562    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:17:27.988752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.988716    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:27.988876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.988808    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:27.988876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:27.988858    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:28.559501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:28.559465    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerStarted","Data":"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"}
Oct 30 07:17:28.559987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:28.559521    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerStarted","Data":"0b91723346ce4cd8fa217924a3bec17caec49cd2966638542495c7f6fb77a5db"}
Oct 30 07:17:28.630103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:28.630048    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podStartSLOduration=1.630034562 podStartE2EDuration="1.630034562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:28.616728665 +0000 UTC m=+1209.049492559" watchObservedRunningTime="2026-10-30 07:17:28.630034562 +0000 UTC m=+1209.062798523"
Oct 30 07:17:28.657461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:28.657429    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:29.521616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.521579    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:29.641852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.641814    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:29.676711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.676680    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:29.905756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.905716    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:17:29.905954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.905932    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy" containerID="cri-o://7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e" gracePeriod=30
Oct 30 07:17:29.909631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.909590    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:17:29.909925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.909890    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy" containerID="cri-o://66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645" gracePeriod=30
Oct 30 07:17:29.916326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.916280    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:17:29.916585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.916538    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy" containerID="cri-o://49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f" gracePeriod=30
Oct 30 07:17:29.920710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.920666    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:17:29.920963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:29.920934    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy" containerID="cri-o://5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90" gracePeriod=30
Oct 30 07:17:30.615547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:30.615513    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:30.616623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:30.616600    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:17:31.142916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.142878    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:31.261563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.261528    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:31.437536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.437444    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:31.452496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.452462    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:31.511162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.511117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:31.679159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:31.679124    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:17:32.902668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:32.902636    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:33.021980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:33.021942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:34.438985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:34.438944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:35.225965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.225024    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:17:35.288556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.288454    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:17:35.292721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292694    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.292846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292753    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.292910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292849    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.292961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292910    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.293020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292959    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.293020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.292999    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.293203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.293114    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.293203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.293170    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.293668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.293632    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.293945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.293917    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\") pod \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\" (UID: \"433c89c0-251b-4a82-8ea2-125f8c3ecd05\") "
Oct 30 07:17:35.294826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.294624    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.294945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.294865    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.294945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.294889    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.294945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.294903    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:35.295128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.295095    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.295228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.295207    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.296517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.296495    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "kube-api-access-7k5f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.297031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.297007    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.297264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.297244    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.300569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.300539    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo" pod "433c89c0-251b-4a82-8ea2-125f8c3ecd05" (UID: "433c89c0-251b-4a82-8ea2-125f8c3ecd05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:35.302427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.302402    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:17:35.306280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.306257    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:17:35.395646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395608    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.395847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395660    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.395847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.395847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395765    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.395847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395799    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.395847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395828    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395874    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395903    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395934    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.395969    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396080    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396118    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396145    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396180    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396216    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396267    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396325    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396428    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396472    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396551    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.396592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396577    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396626    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396747    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396819    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396872    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\") pod \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\" (UID: \"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396926    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\") pod \"7e0608df-6277-4846-bdcb-40ce59f7d918\" (UID: \"7e0608df-6277-4846-bdcb-40ce59f7d918\") "
Oct 30 07:17:35.397018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.396960    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\") pod \"eae54530-42b1-4a0a-bd80-4b53f5038a51\" (UID: \"eae54530-42b1-4a0a-bd80-4b53f5038a51\") "
Oct 30 07:17:35.397678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397648    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.397885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397865    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.397959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397947    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397970    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397987    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k5f4\" (UniqueName: \"kubernetes.io/projected/433c89c0-251b-4a82-8ea2-125f8c3ecd05-kube-api-access-7k5f4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398002    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398016    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398029    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433c89c0-251b-4a82-8ea2-125f8c3ecd05-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398042    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398047    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398056    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433c89c0-251b-4a82-8ea2-125f8c3ecd05-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.398236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.397657    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398255    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398335    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398586    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398711    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398753    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.398938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398916    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.399010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.398921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.399510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.399224    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:35.399510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.399336    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:35.399510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.399413    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:35.399738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.399539    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.401855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.401784    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "kube-api-access-bjh7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.403914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.403888    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:35.405529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405493    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.405627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405552    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405801    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405879    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405901    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6" pod "eae54530-42b1-4a0a-bd80-4b53f5038a51" (UID: "eae54530-42b1-4a0a-bd80-4b53f5038a51"). InnerVolumeSpecName "kube-api-access-5z9j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.405948    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "kube-api-access-cfjrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.406143    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.406333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.406183    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token" pod "7e0608df-6277-4846-bdcb-40ce59f7d918" (UID: "7e0608df-6277-4846-bdcb-40ce59f7d918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:35.406801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.406774    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:35.407204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.407185    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo" pod "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" (UID: "e0085f43-d1ed-4f05-b6ed-aeffdf20cf44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499265    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499313    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfjrd\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-kube-api-access-cfjrd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499329    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499344    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499354    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499363    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499373    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499380    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499388    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499396    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499403    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499412    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499420    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499431    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0608df-6277-4846-bdcb-40ce59f7d918-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499439    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499446    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499454    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9j6\" (UniqueName: \"kubernetes.io/projected/eae54530-42b1-4a0a-bd80-4b53f5038a51-kube-api-access-5z9j6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499462    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae54530-42b1-4a0a-bd80-4b53f5038a51-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499471    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499479    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499487    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjh7h\" (UniqueName: \"kubernetes.io/projected/7e0608df-6277-4846-bdcb-40ce59f7d918-kube-api-access-bjh7h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499494    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499504    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499511    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae54530-42b1-4a0a-bd80-4b53f5038a51-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499520    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.499969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.499528    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0608df-6277-4846-bdcb-40ce59f7d918-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:35.687529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.687495    6738 generic.go:356] "Generic (PLEG): container finished" podID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerID="49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f" exitCode=0
Oct 30 07:17:35.687688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.687605    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"
Oct 30 07:17:35.693092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.693055    6738 generic.go:356] "Generic (PLEG): container finished" podID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerID="7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e" exitCode=0
Oct 30 07:17:35.693262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.693168    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"
Oct 30 07:17:35.695114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.695087    6738 generic.go:356] "Generic (PLEG): container finished" podID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerID="66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645" exitCode=0
Oct 30 07:17:35.695274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.695201    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"
Oct 30 07:17:35.696197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696169    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" event={"ID":"eae54530-42b1-4a0a-bd80-4b53f5038a51","Type":"ContainerDied","Data":"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"}
Oct 30 07:17:35.696349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696227    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh" event={"ID":"eae54530-42b1-4a0a-bd80-4b53f5038a51","Type":"ContainerDied","Data":"cb9c4120d105dacb2be070da097f7f6cc4d85ea8c5334df6ae3196be31e4876f"}
Oct 30 07:17:35.696349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" event={"ID":"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44","Type":"ContainerDied","Data":"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"}
Oct 30 07:17:35.696349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696281    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l" event={"ID":"e0085f43-d1ed-4f05-b6ed-aeffdf20cf44","Type":"ContainerDied","Data":"f538b4632a2fb1de94e57402566869894f84fab39faac0a944fefed1529813c6"}
Oct 30 07:17:35.696349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696320    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" event={"ID":"433c89c0-251b-4a82-8ea2-125f8c3ecd05","Type":"ContainerDied","Data":"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"}
Oct 30 07:17:35.696349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696345    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv" event={"ID":"433c89c0-251b-4a82-8ea2-125f8c3ecd05","Type":"ContainerDied","Data":"6047395e209d15cde52e270529fef8a44d6282de14678f2cef27367e69992658"}
Oct 30 07:17:35.696691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.696375    6738 scope.go:122] "RemoveContainer" containerID="49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"
Oct 30 07:17:35.698493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.698382    6738 generic.go:356] "Generic (PLEG): container finished" podID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerID="5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90" exitCode=0
Oct 30 07:17:35.698493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.698435    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" event={"ID":"7e0608df-6277-4846-bdcb-40ce59f7d918","Type":"ContainerDied","Data":"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"}
Oct 30 07:17:35.698493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.698472    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq" event={"ID":"7e0608df-6277-4846-bdcb-40ce59f7d918","Type":"ContainerDied","Data":"8f8e6deb4af02b67b16db19eaef492fac34c4a4e2a1f911c9b0b3e2d7e2b7f40"}
Oct 30 07:17:35.698668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.698538    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"
Oct 30 07:17:35.725821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.725404    6738 scope.go:122] "RemoveContainer" containerID="49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"
Oct 30 07:17:35.725980 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:35.725867    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f\": container with ID starting with 49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f not found: ID does not exist" containerID="49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"
Oct 30 07:17:35.725980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.725911    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f"} err="failed to get container status \"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f\": rpc error: code = NotFound desc = could not find container \"49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f\": container with ID starting with 49740f4cf5475667666f1224d190a380e24c56824d80bf41c0068506240d954f not found: ID does not exist"
Oct 30 07:17:35.725980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.725934    6738 scope.go:122] "RemoveContainer" containerID="7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"
Oct 30 07:17:35.738728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.738703    6738 scope.go:122] "RemoveContainer" containerID="7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"
Oct 30 07:17:35.739057 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:35.739037    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e\": container with ID starting with 7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e not found: ID does not exist" containerID="7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"
Oct 30 07:17:35.739166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.739063    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e"} err="failed to get container status \"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e\": rpc error: code = NotFound desc = could not find container \"7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e\": container with ID starting with 7001b6b7e05f50c487ebf5f5da1e1b3ebe16c23e55dc2fbc19146f480a985e2e not found: ID does not exist"
Oct 30 07:17:35.739166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.739083    6738 scope.go:122] "RemoveContainer" containerID="66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"
Oct 30 07:17:35.751271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.751249    6738 scope.go:122] "RemoveContainer" containerID="66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"
Oct 30 07:17:35.751628 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:35.751599    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645\": container with ID starting with 66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645 not found: ID does not exist" containerID="66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"
Oct 30 07:17:35.751719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.751635    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645"} err="failed to get container status \"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645\": rpc error: code = NotFound desc = could not find container \"66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645\": container with ID starting with 66a8562418ccf617e41e821f077331fbb5c89c0248277e03091328c6cb7ad645 not found: ID does not exist"
Oct 30 07:17:35.751719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.751657    6738 scope.go:122] "RemoveContainer" containerID="5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"
Oct 30 07:17:35.763152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.763132    6738 scope.go:122] "RemoveContainer" containerID="5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"
Oct 30 07:17:35.763535 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:35.763505    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90\": container with ID starting with 5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90 not found: ID does not exist" containerID="5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"
Oct 30 07:17:35.763615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.763546    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90"} err="failed to get container status \"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90\": rpc error: code = NotFound desc = could not find container \"5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90\": container with ID starting with 5845eb9c154febc96d0e9134399391c767d0edf47b32ac7ac334ed3de8824b90 not found: ID does not exist"
Oct 30 07:17:35.771218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.771179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:35.796813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.796772    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:17:35.801094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.801067    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-82p6l"]
Oct 30 07:17:35.835709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.835663    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:17:35.842159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.842124    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-m56qq"]
Oct 30 07:17:35.905690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.905654    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:17:35.984345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:35.980537    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-7bjzv"]
Oct 30 07:17:36.050377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.042649    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:36.050377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.042744    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:17:36.092333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.091653    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-d6ckh"]
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.348098    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349198    6738 cpu_manager.go:414] "containerMap: removing container" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349312    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349350    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349357    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349451    6738 cpu_manager.go:414] "containerMap: removing container" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349459    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349478    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349487    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349671    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349685    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349700    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" containerName="istio-proxy"
Oct 30 07:17:36.354379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.349709    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" containerName="istio-proxy"
Oct 30 07:17:36.429526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.407721    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:17:36.429526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.407850    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.429526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.424362    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-137ec6-dockercfg-kc82l\""
Oct 30 07:17:36.429526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.425180    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-137ec6\""
Oct 30 07:17:36.495662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.495625    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:17:36.500139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.500118    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.515587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.515555    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:17:36.531870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.531836    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrtb\" (UniqueName: \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.532141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.532087    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4h2c\" (UniqueName: \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.532141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.532130    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.532341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.532282    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.633808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.633772    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4h2c\" (UniqueName: \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.633808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.633810    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.634024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.633859    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.634024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.634003    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrtb\" (UniqueName: \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.634704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.634682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.634764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.634713    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.657715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.657682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4h2c\" (UniqueName: \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\") pod \"payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:36.657842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.657738    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrtb\" (UniqueName: \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\") pod \"payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.758087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.758057    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:36.816841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:36.816809    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:37.065686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.065641    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:17:37.066181 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:37.066159    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307eb1bf_09ed_4d5d_adc6_005e994af3de.slice/crio-f0aed21e7ee147b2263fa55f9813c5744293b8cb38c3ff9356b0f35b84252edb WatchSource:0}: Error finding container f0aed21e7ee147b2263fa55f9813c5744293b8cb38c3ff9356b0f35b84252edb: Status 404 returned error can't find the container with id f0aed21e7ee147b2263fa55f9813c5744293b8cb38c3ff9356b0f35b84252edb
Oct 30 07:17:37.144337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.144256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:37.168797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.168761    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:17:37.171021 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:37.170981    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b35560_06c5_401c_a0b3_ec16999dcbd0.slice/crio-b868bc6b84e6c64ad57b3cf8a1a1f7aded7bcc1ad53e521ff3abb1d184a9e2cf WatchSource:0}: Error finding container b868bc6b84e6c64ad57b3cf8a1a1f7aded7bcc1ad53e521ff3abb1d184a9e2cf: Status 404 returned error can't find the container with id b868bc6b84e6c64ad57b3cf8a1a1f7aded7bcc1ad53e521ff3abb1d184a9e2cf
Oct 30 07:17:37.521116 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:37.521032    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:17:37.692243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.692210    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433c89c0-251b-4a82-8ea2-125f8c3ecd05" path="/var/lib/kubelet/pods/433c89c0-251b-4a82-8ea2-125f8c3ecd05/volumes"
Oct 30 07:17:37.692923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.692902    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0608df-6277-4846-bdcb-40ce59f7d918" path="/var/lib/kubelet/pods/7e0608df-6277-4846-bdcb-40ce59f7d918/volumes"
Oct 30 07:17:37.693554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.693539    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0085f43-d1ed-4f05-b6ed-aeffdf20cf44" path="/var/lib/kubelet/pods/e0085f43-d1ed-4f05-b6ed-aeffdf20cf44/volumes"
Oct 30 07:17:37.694338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.694324    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae54530-42b1-4a0a-bd80-4b53f5038a51" path="/var/lib/kubelet/pods/eae54530-42b1-4a0a-bd80-4b53f5038a51/volumes"
Oct 30 07:17:37.722389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.722353    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" event={"ID":"48b35560-06c5-401c-a0b3-ec16999dcbd0","Type":"ContainerStarted","Data":"5fd9ae74f588f6a922f6a650450244f0d2b2369bb400334eb94ecda1c4f105d3"}
Oct 30 07:17:37.722565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.722422    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:37.722565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.722446    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" event={"ID":"48b35560-06c5-401c-a0b3-ec16999dcbd0","Type":"ContainerStarted","Data":"b868bc6b84e6c64ad57b3cf8a1a1f7aded7bcc1ad53e521ff3abb1d184a9e2cf"}
Oct 30 07:17:37.724060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.724027    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" event={"ID":"307eb1bf-09ed-4d5d-adc6-005e994af3de","Type":"ContainerStarted","Data":"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"}
Oct 30 07:17:37.724186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.724072    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" event={"ID":"307eb1bf-09ed-4d5d-adc6-005e994af3de","Type":"ContainerStarted","Data":"f0aed21e7ee147b2263fa55f9813c5744293b8cb38c3ff9356b0f35b84252edb"}
Oct 30 07:17:37.724186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.724119    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:37.767927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.767865    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" podStartSLOduration=1.7678469209999998 podStartE2EDuration="1.767846921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:37.762787333 +0000 UTC m=+1218.195551226" watchObservedRunningTime="2026-10-30 07:17:37.767846921 +0000 UTC m=+1218.200610818"
Oct 30 07:17:37.804189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:37.804077    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" podStartSLOduration=1.804062946 podStartE2EDuration="1.804062946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:37.803472753 +0000 UTC m=+1218.236236647" watchObservedRunningTime="2026-10-30 07:17:37.804062946 +0000 UTC m=+1218.236826833"
Oct 30 07:17:38.935757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:38.935720    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:40.639950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:40.639905    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:17:40.640576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:40.640104    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy" containerID="cri-o://3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559" gracePeriod=30
Oct 30 07:17:40.652637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:40.652599    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:17:40.652829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:40.652807    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy" containerID="cri-o://983fa0b37fa94b9d33c48e99d698a4ab9a39ab45f0be3d549f5e05cb4d2849a8" gracePeriod=30
Oct 30 07:17:40.770889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:40.770839    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:43.717318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:43.716961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:45.943913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.943883    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:17:45.957487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.957450    6738 generic.go:356] "Generic (PLEG): container finished" podID="33570e67-8a59-4734-81e4-579732636039" containerID="3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559" exitCode=0
Oct 30 07:17:45.957680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.957518    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"
Oct 30 07:17:45.957680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.957546    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" event={"ID":"33570e67-8a59-4734-81e4-579732636039","Type":"ContainerDied","Data":"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"}
Oct 30 07:17:45.957680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.957612    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857" event={"ID":"33570e67-8a59-4734-81e4-579732636039","Type":"ContainerDied","Data":"3121842a0d01145881839f2ffbee34e9e4317f0e3383f45b48f3397e0864eb5a"}
Oct 30 07:17:45.957680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.957643    6738 scope.go:122] "RemoveContainer" containerID="3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"
Oct 30 07:17:45.960098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.959725    6738 generic.go:356] "Generic (PLEG): container finished" podID="4256c405-e04d-42d3-b223-345ca45f754f" containerID="983fa0b37fa94b9d33c48e99d698a4ab9a39ab45f0be3d549f5e05cb4d2849a8" exitCode=0
Oct 30 07:17:45.960098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.959867    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" event={"ID":"4256c405-e04d-42d3-b223-345ca45f754f","Type":"ContainerDied","Data":"983fa0b37fa94b9d33c48e99d698a4ab9a39ab45f0be3d549f5e05cb4d2849a8"}
Oct 30 07:17:45.991159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.991133    6738 scope.go:122] "RemoveContainer" containerID="3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"
Oct 30 07:17:45.992184 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:45.992145    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559\": container with ID starting with 3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559 not found: ID does not exist" containerID="3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"
Oct 30 07:17:45.992343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.992191    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559"} err="failed to get container status \"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559\": rpc error: code = NotFound desc = could not find container \"3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559\": container with ID starting with 3708c0f0d9b341faad83d47929e23550e4af82db2384c10a87cf80b9141f0559 not found: ID does not exist"
Oct 30 07:17:45.995420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:45.995401    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:17:46.084008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.083970    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084043    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084152    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084212    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084251    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084334    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084371    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084432    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084482    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084537    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084590    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084641    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084723    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084752    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\") pod \"4256c405-e04d-42d3-b223-345ca45f754f\" (UID: \"4256c405-e04d-42d3-b223-345ca45f754f\") "
Oct 30 07:17:46.084843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084780    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084858    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.084993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084895    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.084993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084962    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.085134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.084997    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\") pod \"33570e67-8a59-4734-81e4-579732636039\" (UID: \"33570e67-8a59-4734-81e4-579732636039\") "
Oct 30 07:17:46.085134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085020    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.085252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085223    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.085356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085279    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.085662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085641    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.085737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085666    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.085737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085681    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.085737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085695    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.085896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085825    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.085957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.085911    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.086086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.086057    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.086235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.086202    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:46.086569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.086522    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:46.086760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.086731    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.088849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.088746    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "kube-api-access-7mkjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:46.088980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.088816    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:46.089797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.089753    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:46.090449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.090428    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "kube-api-access-f2q57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:46.091459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.091426    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.093113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.092487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:46.093331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.093278    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo" pod "4256c405-e04d-42d3-b223-345ca45f754f" (UID: "4256c405-e04d-42d3-b223-345ca45f754f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:46.101399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.101364    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy" pod "33570e67-8a59-4734-81e4-579732636039" (UID: "33570e67-8a59-4734-81e4-579732636039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:46.189624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189531    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189572    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2q57\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-kube-api-access-f2q57\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189587    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33570e67-8a59-4734-81e4-579732636039-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189600    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189615    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33570e67-8a59-4734-81e4-579732636039-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189631    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkjl\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-kube-api-access-7mkjl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189645    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256c405-e04d-42d3-b223-345ca45f754f-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189656    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189669    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256c405-e04d-42d3-b223-345ca45f754f-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189680    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33570e67-8a59-4734-81e4-579732636039-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189693    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189705    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256c405-e04d-42d3-b223-345ca45f754f-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189717    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33570e67-8a59-4734-81e4-579732636039-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.189929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.189728    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256c405-e04d-42d3-b223-345ca45f754f-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.308506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.308473    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:17:46.317149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.317118    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-8r857"]
Oct 30 07:17:46.389917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.389881    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:46.970451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.970374    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"
Oct 30 07:17:46.970451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.970383    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h" event={"ID":"4256c405-e04d-42d3-b223-345ca45f754f","Type":"ContainerDied","Data":"0b4c7c1044965bb22ed04cb95490e38e4820b3e728cf75866565847766cb34b0"}
Oct 30 07:17:46.970451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:46.970447    6738 scope.go:122] "RemoveContainer" containerID="983fa0b37fa94b9d33c48e99d698a4ab9a39ab45f0be3d549f5e05cb4d2849a8"
Oct 30 07:17:47.024746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:47.024708    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:17:47.036727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:47.036693    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-pzc7h"]
Oct 30 07:17:47.690682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:47.690649    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33570e67-8a59-4734-81e4-579732636039" path="/var/lib/kubelet/pods/33570e67-8a59-4734-81e4-579732636039/volumes"
Oct 30 07:17:47.691242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:47.691228    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4256c405-e04d-42d3-b223-345ca45f754f" path="/var/lib/kubelet/pods/4256c405-e04d-42d3-b223-345ca45f754f/volumes"
Oct 30 07:17:48.306395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:48.306354    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:48.744208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:48.744172    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:17:48.744392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:48.744259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:17:50.160283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:50.160230    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:51.085574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.085535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:51.538569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.538530    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:17:51.539535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539507    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy"
Oct 30 07:17:51.539535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539533    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy"
Oct 30 07:17:51.539727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539596    6738 cpu_manager.go:414] "containerMap: removing container" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy"
Oct 30 07:17:51.539727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539603    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy"
Oct 30 07:17:51.539845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539755    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4256c405-e04d-42d3-b223-345ca45f754f" containerName="istio-proxy"
Oct 30 07:17:51.539845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.539777    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="33570e67-8a59-4734-81e4-579732636039" containerName="istio-proxy"
Oct 30 07:17:51.545253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.545230    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.575416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.575386    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:17:51.583166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.583130    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f7467d-openshift-default-dockercfg-m8zjr\""
Oct 30 07:17:51.588954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.588893    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.588977    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589019    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589111    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589139    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589161    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589190    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589285    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5f5s\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.589610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.589354    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.676036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.675999    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:51.690409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690371    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5f5s\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690436    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690630    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690721    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690759    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690824    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690850    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690870    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690876    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.690971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.690898    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.691134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.691032    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.691134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.691121    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.691269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.691255    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.691487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.691468    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.693013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.692988    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.693237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.693219    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.714530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.714501    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.714683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.714590    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5f5s\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\") pod \"e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") " pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:51.910539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.910498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:51.917681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:51.917650    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:52.001857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.001816    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:17:52.002119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.002101    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:17:52.002274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.002260    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:17:52.002625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.002581    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy" containerID="cri-o://9589bd1e37c172faa721f5a0301b96a8c34a21c33552aaa88df4e6ed19462baf" gracePeriod=30
Oct 30 07:17:52.009064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.009003    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy" containerID="cri-o://67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1" gracePeriod=30
Oct 30 07:17:52.009498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.009468    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy" containerID="cri-o://d37581fad5a5002c481fe0f4f74118a8fdc9609166f7fcb1fd2df25cdcb59214" gracePeriod=30
Oct 30 07:17:52.010144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.010121    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:17:52.010611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.010585    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy" containerID="cri-o://b621dbc3bd3d10727a7cd8ff3fc5a0d2733e4677d290e34f0f1988be2cf97556" gracePeriod=30
Oct 30 07:17:52.642756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.642715    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:52.786257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.786159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:52.890322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.889436    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"]
Oct 30 07:17:52.901018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.900986    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4e42c8-hqpq9"]
Oct 30 07:17:52.925684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.925654    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:17:52.926247 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:52.926215    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e77782b_d1c7_4b84_80e4_11b2e92d07a0.slice/crio-41cf767dade6d295a4194147ebd5064e252bf335a67c11f0a81299ff75fa7726 WatchSource:0}: Error finding container 41cf767dade6d295a4194147ebd5064e252bf335a67c11f0a81299ff75fa7726: Status 404 returned error can't find the container with id 41cf767dade6d295a4194147ebd5064e252bf335a67c11f0a81299ff75fa7726
Oct 30 07:17:52.929465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.929427    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:52.929566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.929500    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:52.929566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.929530    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:17:52.950501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:52.950466    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:53.142464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.142426    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerStarted","Data":"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"}
Oct 30 07:17:53.142642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.142482    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerStarted","Data":"41cf767dade6d295a4194147ebd5064e252bf335a67c11f0a81299ff75fa7726"}
Oct 30 07:17:53.191863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.191800    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podStartSLOduration=2.191780906 podStartE2EDuration="2.191780906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:53.188066308 +0000 UTC m=+1233.620830200" watchObservedRunningTime="2026-10-30 07:17:53.191780906 +0000 UTC m=+1233.624544866"
Oct 30 07:17:53.354010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.353918    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:53.690780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.690743    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843afc2f-4512-4b0c-bffe-aafdae44bbfa" path="/var/lib/kubelet/pods/843afc2f-4512-4b0c-bffe-aafdae44bbfa/volumes"
Oct 30 07:17:53.936271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:53.936234    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:54.941841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:54.941799    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:55.151786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.151749    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:55.158791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.158760    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:55.160093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.160069    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:17:55.303342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.303208    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:55.762171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.762139    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:55.923804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:55.923765    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:57.194018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.193976    6738 generic.go:356] "Generic (PLEG): container finished" podID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerID="d37581fad5a5002c481fe0f4f74118a8fdc9609166f7fcb1fd2df25cdcb59214" exitCode=0
Oct 30 07:17:57.194467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.194167    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" event={"ID":"5ec57481-f6e1-4866-9503-603d9c9bba1b","Type":"ContainerDied","Data":"d37581fad5a5002c481fe0f4f74118a8fdc9609166f7fcb1fd2df25cdcb59214"}
Oct 30 07:17:57.196633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.196606    6738 generic.go:356] "Generic (PLEG): container finished" podID="1b217882-7f4a-4546-aaff-79d93995baf6" containerID="9589bd1e37c172faa721f5a0301b96a8c34a21c33552aaa88df4e6ed19462baf" exitCode=0
Oct 30 07:17:57.196796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.196770    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" event={"ID":"1b217882-7f4a-4546-aaff-79d93995baf6","Type":"ContainerDied","Data":"9589bd1e37c172faa721f5a0301b96a8c34a21c33552aaa88df4e6ed19462baf"}
Oct 30 07:17:57.199616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.199592    6738 generic.go:356] "Generic (PLEG): container finished" podID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerID="b621dbc3bd3d10727a7cd8ff3fc5a0d2733e4677d290e34f0f1988be2cf97556" exitCode=0
Oct 30 07:17:57.200364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.200338    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" event={"ID":"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c","Type":"ContainerDied","Data":"b621dbc3bd3d10727a7cd8ff3fc5a0d2733e4677d290e34f0f1988be2cf97556"}
Oct 30 07:17:57.303060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.303031    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:17:57.389782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.389755    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:17:57.427451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427417    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.427609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427471    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.427609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427521    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.427719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427629    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.427787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427727    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.427787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.427910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427808    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.427910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427834    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.427910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427868    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.428068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427914    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.428068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.427967    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.428068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428000    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.428068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.428265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428070    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.428265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428108    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.428265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428166    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.428265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428242    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\") pod \"1b217882-7f4a-4546-aaff-79d93995baf6\" (UID: \"1b217882-7f4a-4546-aaff-79d93995baf6\") "
Oct 30 07:17:57.428493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.428416    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\") pod \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\" (UID: \"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c\") "
Oct 30 07:17:57.430262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.430186    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:57.430645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.430529    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:57.430992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.430815    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:17:57.431743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431417    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.431844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431787    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.431946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431848    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.431946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431899    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.431946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431924    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:57.432062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.431949    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.432643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.432562    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.433417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.433334    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.433417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.433393    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.433635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.433605    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.434795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.434767    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "kube-api-access-ssszf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.435048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.434982    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.435048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.435012    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "kube-api-access-lg5w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.436104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.436069    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:57.436904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.436872    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token" pod "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" (UID: "a6ccb3aa-c165-4a62-b007-c35ba0c9b00c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.437175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.437153    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy" pod "1b217882-7f4a-4546-aaff-79d93995baf6" (UID: "1b217882-7f4a-4546-aaff-79d93995baf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.494840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.494805    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:17:57.529932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.529857    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.529985    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530041    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530089    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530151    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530252    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530323    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530362    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.530583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530559    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530623    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\") pod \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\" (UID: \"f3ea3421-c4fe-44cd-80e4-b9e19507209f\") "
Oct 30 07:17:57.530723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530703    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.530777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530731    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:57.530979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530954    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.531215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.530974    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.531707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.531686    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.531826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.531813    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.531930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.531919    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b217882-7f4a-4546-aaff-79d93995baf6-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532016    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b217882-7f4a-4546-aaff-79d93995baf6-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532107    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5w6\" (UniqueName: \"kubernetes.io/projected/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-kube-api-access-lg5w6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532201    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532369    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532532    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532652    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532745    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532848    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.532942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.532932    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533016    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533094    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533183    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533276    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533437    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533460    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533473    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b217882-7f4a-4546-aaff-79d93995baf6-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533481    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533488    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533534    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssszf\" (UniqueName: \"kubernetes.io/projected/1b217882-7f4a-4546-aaff-79d93995baf6-kube-api-access-ssszf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533551    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533566    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.533831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533627    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:57.533831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533643    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "kube-api-access-hzw52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.533928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.533883    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy" pod "f3ea3421-c4fe-44cd-80e4-b9e19507209f" (UID: "f3ea3421-c4fe-44cd-80e4-b9e19507209f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.535637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.535591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:57.634231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634191    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634289    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634394    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634427    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634489    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634511    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634543    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634580    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634620    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634651    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\") pod \"5ec57481-f6e1-4866-9503-603d9c9bba1b\" (UID: \"5ec57481-f6e1-4866-9503-603d9c9bba1b\") "
Oct 30 07:17:57.634958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634842    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.634958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.634857    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.635327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635269    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:57.635445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635410    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.635662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635636    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635665    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635681    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635696    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635708    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635721    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzw52\" (UniqueName: \"kubernetes.io/projected/f3ea3421-c4fe-44cd-80e4-b9e19507209f-kube-api-access-hzw52\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635737    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ea3421-c4fe-44cd-80e4-b9e19507209f-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635750    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.635775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.635765    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec57481-f6e1-4866-9503-603d9c9bba1b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.636772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.636750    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:57.636925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.636906    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:57.637019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.636947    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "kube-api-access-jkvgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.637019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.636976    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token" pod "5ec57481-f6e1-4866-9503-603d9c9bba1b" (UID: "5ec57481-f6e1-4866-9503-603d9c9bba1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:57.671511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.671472    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:57.709379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.709354    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:17:57.709987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.709972    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy"
Oct 30 07:17:57.710033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.709989    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy"
Oct 30 07:17:57.710033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710028    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy"
Oct 30 07:17:57.710094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710033    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy"
Oct 30 07:17:57.710094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710044    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy"
Oct 30 07:17:57.710094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710048    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy"
Oct 30 07:17:57.710094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710056    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy"
Oct 30 07:17:57.710094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710060    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy"
Oct 30 07:17:57.710261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710150    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" containerName="istio-proxy"
Oct 30 07:17:57.710261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710166    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" containerName="istio-proxy"
Oct 30 07:17:57.710261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710171    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerName="istio-proxy"
Oct 30 07:17:57.710261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.710178    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" containerName="istio-proxy"
Oct 30 07:17:57.715239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.715220    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.732422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.732391    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f7467d\""
Oct 30 07:17:57.732649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.732626    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f7467d-dockercfg-kndjr\""
Oct 30 07:17:57.734317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.734275    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:17:57.737821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.737796    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.737928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.737824    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.737928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.737838    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvgd\" (UniqueName: \"kubernetes.io/projected/5ec57481-f6e1-4866-9503-603d9c9bba1b-kube-api-access-jkvgd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.737928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.737852    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec57481-f6e1-4866-9503-603d9c9bba1b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:17:57.784842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.784743    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:17:57.789405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.789377    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:57.789862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.789823    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:57.807863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.807835    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:17:57.848415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.848378    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4qh\" (UniqueName: \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:57.848852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.848818    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ngf\" (UniqueName: \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.849195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.849172    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.849864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.849839    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:57.955258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.955209    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:57.955861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.955829    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4qh\" (UniqueName: \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:57.956101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.956086    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ngf\" (UniqueName: \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.956287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.956267    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.957459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.957434    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:57.960281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:57.960256    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:58.017880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.017838    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4qh\" (UniqueName: \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\") pod \"payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:58.026229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.026190    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ngf\" (UniqueName: \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:58.034314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.030566    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:58.105505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.105456    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:58.216329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.216263    6738 generic.go:356] "Generic (PLEG): container finished" podID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" containerID="67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1" exitCode=0
Oct 30 07:17:58.216700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.216469    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" event={"ID":"f3ea3421-c4fe-44cd-80e4-b9e19507209f","Type":"ContainerDied","Data":"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"}
Oct 30 07:17:58.216700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.216530    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q" event={"ID":"f3ea3421-c4fe-44cd-80e4-b9e19507209f","Type":"ContainerDied","Data":"a42feec5e84ceae32516ac1978c8e163ca18dead9d358b8c7864a518d3799eda"}
Oct 30 07:17:58.216700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.216561    6738 scope.go:122] "RemoveContainer" containerID="67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"
Oct 30 07:17:58.216820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.216746    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"
Oct 30 07:17:58.218992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.218974    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"
Oct 30 07:17:58.219883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.219856    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7" event={"ID":"5ec57481-f6e1-4866-9503-603d9c9bba1b","Type":"ContainerDied","Data":"d7170e1ca38a2c302fdd43e8d8bf4340f94473c631304b420e3448002823613a"}
Oct 30 07:17:58.221384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.221361    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v" event={"ID":"1b217882-7f4a-4546-aaff-79d93995baf6","Type":"ContainerDied","Data":"f1bdbb7ab93d61e5c8c00aec95acb1a8691c402217b07daa52507942ffb01d04"}
Oct 30 07:17:58.221502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.221487    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"
Oct 30 07:17:58.230559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.230503    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5" event={"ID":"a6ccb3aa-c165-4a62-b007-c35ba0c9b00c","Type":"ContainerDied","Data":"b6f0553008e922a94b4a61b23bc66df4a385ae5d9a4a30fa069c8992d8741d9a"}
Oct 30 07:17:58.230559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.230549    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"
Oct 30 07:17:58.235466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.235444    6738 scope.go:122] "RemoveContainer" containerID="67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"
Oct 30 07:17:58.235819 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:17:58.235788    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1\": container with ID starting with 67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1 not found: ID does not exist" containerID="67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"
Oct 30 07:17:58.235899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.235834    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1"} err="failed to get container status \"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1\": rpc error: code = NotFound desc = could not find container \"67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1\": container with ID starting with 67e679d29c1bdb0032d951c98a8a38bb6470c2c2ab0c62df0577e26d1955cdb1 not found: ID does not exist"
Oct 30 07:17:58.235899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.235857    6738 scope.go:122] "RemoveContainer" containerID="d37581fad5a5002c481fe0f4f74118a8fdc9609166f7fcb1fd2df25cdcb59214"
Oct 30 07:17:58.249108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.249084    6738 scope.go:122] "RemoveContainer" containerID="9589bd1e37c172faa721f5a0301b96a8c34a21c33552aaa88df4e6ed19462baf"
Oct 30 07:17:58.271538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.271512    6738 scope.go:122] "RemoveContainer" containerID="b621dbc3bd3d10727a7cd8ff3fc5a0d2733e4677d290e34f0f1988be2cf97556"
Oct 30 07:17:58.277027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.276998    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:17:58.290745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.290707    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-wgnq7"]
Oct 30 07:17:58.315852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.315763    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:17:58.320381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.320353    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-b2kg5"]
Oct 30 07:17:58.356760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.356708    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:17:58.360109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.360081    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-qb99q"]
Oct 30 07:17:58.393098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.393056    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:17:58.393428 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:58.393399    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744bc65f_efcd_4868_a43a_904c4cadb8aa.slice/crio-19bbe58d08bb2304918f1bd67ea7f0f779707999674354d064aed1ced58efe4b WatchSource:0}: Error finding container 19bbe58d08bb2304918f1bd67ea7f0f779707999674354d064aed1ced58efe4b: Status 404 returned error can't find the container with id 19bbe58d08bb2304918f1bd67ea7f0f779707999674354d064aed1ced58efe4b
Oct 30 07:17:58.399538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.399514    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:17:58.404832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.404807    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-hjd9v"]
Oct 30 07:17:58.448718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:58.448692    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:17:58.457780 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:17:58.457750    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6937871_1215_4f8a_a1c2_3886b26f7021.slice/crio-f2fe49c2eb90ed515dee94771930b9467ea48e4787640e883b4ef257fd0156a2 WatchSource:0}: Error finding container f2fe49c2eb90ed515dee94771930b9467ea48e4787640e883b4ef257fd0156a2: Status 404 returned error can't find the container with id f2fe49c2eb90ed515dee94771930b9467ea48e4787640e883b4ef257fd0156a2
Oct 30 07:17:59.019768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.019727    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:17:59.243092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.243055    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" event={"ID":"744bc65f-efcd-4868-a43a-904c4cadb8aa","Type":"ContainerStarted","Data":"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"}
Oct 30 07:17:59.243556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.243110    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" event={"ID":"744bc65f-efcd-4868-a43a-904c4cadb8aa","Type":"ContainerStarted","Data":"19bbe58d08bb2304918f1bd67ea7f0f779707999674354d064aed1ced58efe4b"}
Oct 30 07:17:59.243556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.243140    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:17:59.246733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.246706    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" event={"ID":"e6937871-1215-4f8a-a1c2-3886b26f7021","Type":"ContainerStarted","Data":"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"}
Oct 30 07:17:59.246884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.246749    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" event={"ID":"e6937871-1215-4f8a-a1c2-3886b26f7021","Type":"ContainerStarted","Data":"f2fe49c2eb90ed515dee94771930b9467ea48e4787640e883b4ef257fd0156a2"}
Oct 30 07:17:59.246884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.246787    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:17:59.287029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.286926    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" podStartSLOduration=2.28691231 podStartE2EDuration="2.28691231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:59.283776006 +0000 UTC m=+1239.716539896" watchObservedRunningTime="2026-10-30 07:17:59.28691231 +0000 UTC m=+1239.719676202"
Oct 30 07:17:59.324052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.323997    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" podStartSLOduration=2.323983013 podStartE2EDuration="2.323983013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:59.323884243 +0000 UTC m=+1239.756648136" watchObservedRunningTime="2026-10-30 07:17:59.323983013 +0000 UTC m=+1239.756746906"
Oct 30 07:17:59.693757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.693712    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b217882-7f4a-4546-aaff-79d93995baf6" path="/var/lib/kubelet/pods/1b217882-7f4a-4546-aaff-79d93995baf6/volumes"
Oct 30 07:17:59.694584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.694558    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec57481-f6e1-4866-9503-603d9c9bba1b" path="/var/lib/kubelet/pods/5ec57481-f6e1-4866-9503-603d9c9bba1b/volumes"
Oct 30 07:17:59.695337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.695314    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ccb3aa-c165-4a62-b007-c35ba0c9b00c" path="/var/lib/kubelet/pods/a6ccb3aa-c165-4a62-b007-c35ba0c9b00c/volumes"
Oct 30 07:17:59.696530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:17:59.696507    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ea3421-c4fe-44cd-80e4-b9e19507209f" path="/var/lib/kubelet/pods/f3ea3421-c4fe-44cd-80e4-b9e19507209f/volumes"
Oct 30 07:18:00.036167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.036060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:00.176150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.176117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:00.398346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.398282    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:18:00.398774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.398519    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy" containerID="cri-o://1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68" gracePeriod=30
Oct 30 07:18:00.436490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.436451    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:00.746752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:00.746658    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:01.036677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:01.036572    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"]
Oct 30 07:18:01.040365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:01.040330    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee2c980-cdb9g"]
Oct 30 07:18:01.692173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:01.692135    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14408016-3c28-4a4e-bc94-e3122d4ad5d1" path="/var/lib/kubelet/pods/14408016-3c28-4a4e-bc94-e3122d4ad5d1/volumes"
Oct 30 07:18:02.346016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.345978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:02.773846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.773813    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:18:02.782250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.782222    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.798314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.798165    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a51baa-openshift-default-dockercfg-gh6wr\""
Oct 30 07:18:02.815903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.815861    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:18:02.837041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837003    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837042    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837072    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837205    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837248    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837339    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837365    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvnj\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837409    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.837516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.837443    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.853985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.853927    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:02.939026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.938988    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939045    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939098    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939135    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939245    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939276    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvnj\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939365    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.939516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.939429    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.940278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.940250    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.941020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.940458    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.941174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.940697    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.941416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.941384    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.941893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.941871    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.943922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.943892    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.956939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.955666    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:02.986320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:02.980991    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvnj\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:03.009316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.009001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:03.013318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.009576    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\") pod \"e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") " pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:03.116371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.106546    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:03.322133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.322093    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:03.535945 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:03.535913    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1871aeae_1fad_4a35_ad48_4f4c82111d52.slice/crio-4f89c82200f211552da92279e76453ac8fe8f412ac456baf0eacd4aceb05fe57 WatchSource:0}: Error finding container 4f89c82200f211552da92279e76453ac8fe8f412ac456baf0eacd4aceb05fe57: Status 404 returned error can't find the container with id 4f89c82200f211552da92279e76453ac8fe8f412ac456baf0eacd4aceb05fe57
Oct 30 07:18:03.535945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.535930    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:18:03.538829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.538795    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:03.538932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.538866    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:03.538932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:03.538902    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:04.327758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:04.327720    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerStarted","Data":"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"}
Oct 30 07:18:04.328149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:04.327814    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerStarted","Data":"4f89c82200f211552da92279e76453ac8fe8f412ac456baf0eacd4aceb05fe57"}
Oct 30 07:18:04.387579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:04.387525    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podStartSLOduration=2.387512154 podStartE2EDuration="2.387512154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:04.384779902 +0000 UTC m=+1244.817543783" watchObservedRunningTime="2026-10-30 07:18:04.387512154 +0000 UTC m=+1244.820276111"
Oct 30 07:18:05.107178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.107143    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:05.113252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.113226    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:05.152309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.152242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:05.343640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.343607    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:05.344677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.344655    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:18:05.696792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.696757    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:18:05.734965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.734933    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735017    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735067    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735095    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735171    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735209    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:18:05.735335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735230    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735317    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735384    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:18:05.735535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735450    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735489    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\") pod \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\" (UID: \"4ecbecbb-2144-45db-8df9-47357b2dbf9f\") "
Oct 30 07:18:05.735535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735517    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:18:05.735750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735544    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:18:05.735882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.735857    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:18:05.737953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.737924    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.738045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.737957    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.738045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.737971    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.738045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.737985    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.738045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.737998    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.738533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.738485    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:05.738617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.738552    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "kube-api-access-cmth5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:05.738670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.738635    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:18:05.738770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.738752    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy" pod "4ecbecbb-2144-45db-8df9-47357b2dbf9f" (UID: "4ecbecbb-2144-45db-8df9-47357b2dbf9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:18:05.839243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.839198    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.839243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.839233    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.839243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.839244    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:05.839243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:05.839253    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmth5\" (UniqueName: \"kubernetes.io/projected/4ecbecbb-2144-45db-8df9-47357b2dbf9f-kube-api-access-cmth5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:06.353515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.353482    6738 generic.go:356] "Generic (PLEG): container finished" podID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerID="1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68" exitCode=0
Oct 30 07:18:06.353987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.353539    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"
Oct 30 07:18:06.353987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.353568    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerDied","Data":"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"}
Oct 30 07:18:06.353987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.353628    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw" event={"ID":"4ecbecbb-2144-45db-8df9-47357b2dbf9f","Type":"ContainerDied","Data":"ed18d3aed87deba7157a7998b98342121284198a7d79124b4c14abe67426f498"}
Oct 30 07:18:06.353987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.353652    6738 scope.go:122] "RemoveContainer" containerID="1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"
Oct 30 07:18:06.365480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.365459    6738 scope.go:122] "RemoveContainer" containerID="2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"
Oct 30 07:18:06.376091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.376066    6738 scope.go:122] "RemoveContainer" containerID="1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"
Oct 30 07:18:06.376494 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:18:06.376475    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68\": container with ID starting with 1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68 not found: ID does not exist" containerID="1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"
Oct 30 07:18:06.376555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.376502    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68"} err="failed to get container status \"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68\": rpc error: code = NotFound desc = could not find container \"1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68\": container with ID starting with 1534abc0bccf9ba0e3bb6af54c444d50069224ae5051a32806ebbc18f46f1c68 not found: ID does not exist"
Oct 30 07:18:06.376555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.376519    6738 scope.go:122] "RemoveContainer" containerID="2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"
Oct 30 07:18:06.376940 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:18:06.376919    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b\": container with ID starting with 2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b not found: ID does not exist" containerID="2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"
Oct 30 07:18:06.376986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.376946    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b"} err="failed to get container status \"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b\": rpc error: code = NotFound desc = could not find container \"2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b\": container with ID starting with 2d7d8d48bf5eefdd225091b80eac46c8fce42f3768b11bec845ba1f0c233475b not found: ID does not exist"
Oct 30 07:18:06.414738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.414701    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:18:06.427181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.427151    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4e42c8-openshift-default-67dbb6c9f-mswpw"]
Oct 30 07:18:06.836458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.836414    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:06.964712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:06.964675    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:07.383671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:07.383634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:07.530204 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:18:07.530156    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:18:07.693737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:07.693654    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" path="/var/lib/kubelet/pods/4ecbecbb-2144-45db-8df9-47357b2dbf9f/volumes"
Oct 30 07:18:08.654271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.654231    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:08.775361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.775322    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:18:08.776089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776065    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.776089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776086    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.776248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776098    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.776248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776104    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.776248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776212    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.776248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.776221    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecbecbb-2144-45db-8df9-47357b2dbf9f" containerName="istio-proxy"
Oct 30 07:18:08.781353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.781331    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.783031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.782998    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:08.783906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.783883    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9qp\" (UniqueName: \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.784137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.784114    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.797894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.797864    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:18:08.806366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.806335    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a51baa\""
Oct 30 07:18:08.806572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.806413    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a51baa-dockercfg-jxxld\""
Oct 30 07:18:08.841995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.841961    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:18:08.846259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.846236    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:08.865639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.865603    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:18:08.886538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.886486    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.886756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.886596    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:08.886756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.886687    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhvm\" (UniqueName: \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:08.886882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.886759    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9qp\" (UniqueName: \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.958373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.896118    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.958693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.921055    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9qp\" (UniqueName: \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\") pod \"payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:08.998399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.995483    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:08.998399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.995602    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhvm\" (UniqueName: \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:08.998399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:08.997648    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:09.120645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:09.120393    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:09.120645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:09.120474    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhvm\" (UniqueName: \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\") pod \"payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:09.162449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:09.162417    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:09.460747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:09.460711    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:18:09.461043 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:09.461019    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213d2a9c_fee7_401d_acc5_df917039a58a.slice/crio-4719dbbddf4ad51e3292b095a633c5e6472a8994e9b432dbce54bd6a1828b664 WatchSource:0}: Error finding container 4719dbbddf4ad51e3292b095a633c5e6472a8994e9b432dbce54bd6a1828b664: Status 404 returned error can't find the container with id 4719dbbddf4ad51e3292b095a633c5e6472a8994e9b432dbce54bd6a1828b664
Oct 30 07:18:09.493047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:09.493021    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:18:09.505626 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:09.505598    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397d80c5_c41d_46a2_b09f_db22e8cfec3f.slice/crio-355d5db40da5e7652188eddc9e074a290976bb4aef0de96666b250794534a73e WatchSource:0}: Error finding container 355d5db40da5e7652188eddc9e074a290976bb4aef0de96666b250794534a73e: Status 404 returned error can't find the container with id 355d5db40da5e7652188eddc9e074a290976bb4aef0de96666b250794534a73e
Oct 30 07:18:10.045851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.045809    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:10.295526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.295496    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:18:10.295916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.295866    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:18:10.394372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.394317    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" event={"ID":"397d80c5-c41d-46a2-b09f-db22e8cfec3f","Type":"ContainerStarted","Data":"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"}
Oct 30 07:18:10.394572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.394401    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" event={"ID":"397d80c5-c41d-46a2-b09f-db22e8cfec3f","Type":"ContainerStarted","Data":"355d5db40da5e7652188eddc9e074a290976bb4aef0de96666b250794534a73e"}
Oct 30 07:18:10.394572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.394524    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:10.396661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.396629    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" event={"ID":"213d2a9c-fee7-401d-acc5-df917039a58a","Type":"ContainerStarted","Data":"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"}
Oct 30 07:18:10.396842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.396690    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:10.396842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.396716    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" event={"ID":"213d2a9c-fee7-401d-acc5-df917039a58a","Type":"ContainerStarted","Data":"4719dbbddf4ad51e3292b095a633c5e6472a8994e9b432dbce54bd6a1828b664"}
Oct 30 07:18:10.438486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.438434    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" podStartSLOduration=2.438419105 podStartE2EDuration="2.438419105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:10.435772047 +0000 UTC m=+1250.868535940" watchObservedRunningTime="2026-10-30 07:18:10.438419105 +0000 UTC m=+1250.871182998"
Oct 30 07:18:10.480194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:10.480138    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" podStartSLOduration=2.480122583 podStartE2EDuration="2.480122583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:10.476706893 +0000 UTC m=+1250.909470786" watchObservedRunningTime="2026-10-30 07:18:10.480122583 +0000 UTC m=+1250.912886499"
Oct 30 07:18:12.072314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:12.054355    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"]
Oct 30 07:18:12.099380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:12.090768    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a49e384-bltmg"]
Oct 30 07:18:12.561006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:12.560965    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:13.701685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:13.701650    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2" path="/var/lib/kubelet/pods/c5fb4366-c3f5-4f6c-a94c-bfbc80d505c2/volumes"
Oct 30 07:18:14.491731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:14.491691    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:16.419994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:16.419960    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:18.350662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:18.350621    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:20.213990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:20.213884    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:20.327862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:20.327826    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:20.446570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:20.444800    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:21.458317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:21.457991    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:18:21.476780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:21.475543    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:18:22.244759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:22.244715    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:23.455993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.455959    6738 scope.go:122] "RemoveContainer" containerID="36126248fd89926333ee00c86017de7b63ad6b15751bd78c3a0b521440e5fbbc"
Oct 30 07:18:23.466437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.466405    6738 scope.go:122] "RemoveContainer" containerID="f5a55426ed23350028d3db4c8e611f635bb1931d37a885afccb7da6abf545f50"
Oct 30 07:18:23.476245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.476221    6738 scope.go:122] "RemoveContainer" containerID="f3090bc07c323fbc35ffd2a498d9d8e059fcfbbaa9d50c7d58d3005585dcbc49"
Oct 30 07:18:23.487000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.486977    6738 scope.go:122] "RemoveContainer" containerID="ca5e37d8efd8793f7a5ba8565f316c248830f9439cc586acb28d71647a6408e5"
Oct 30 07:18:23.497142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.497117    6738 scope.go:122] "RemoveContainer" containerID="d4d9ba8ff7fd0397359ed070b86260a0f90911821a975170fcf05a32774e1325"
Oct 30 07:18:23.507412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.507389    6738 scope.go:122] "RemoveContainer" containerID="a26fdecf902cd94521ccce9ef8ff5a80c94d771c2637933455134efa7e56c75d"
Oct 30 07:18:23.517197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.517168    6738 scope.go:122] "RemoveContainer" containerID="7163acfe7f72d0e59ee5265369b47411a7b1c7fc12098d48308d2dde79be2734"
Oct 30 07:18:23.526939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.526916    6738 scope.go:122] "RemoveContainer" containerID="566d0b177876bc6d0d473e4f0621d5f1a720b3a22c1203d4f7850c9664a18009"
Oct 30 07:18:23.536477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.536456    6738 scope.go:122] "RemoveContainer" containerID="2d97330bb3673cb6a2bb36a1f723635648705e9435c6d1ecabd22f02acfda506"
Oct 30 07:18:23.596186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.596159    6738 generic.go:356] "Generic (PLEG): container finished" podID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerID="534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f" exitCode=0
Oct 30 07:18:23.596346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.596244    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerDied","Data":"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"}
Oct 30 07:18:23.596805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.596784    6738 scope.go:122] "RemoveContainer" containerID="534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"
Oct 30 07:18:23.598783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.598746    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:23.598890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.598850    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:23.598952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:23.598909    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:24.392371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:24.386832    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:24.613622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:24.613585    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerStarted","Data":"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"}
Oct 30 07:18:24.657776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:24.657703    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:18:24.657928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:24.657796    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:24.657928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:24.657829    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:25.657048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:25.657017    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:25.657564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:25.657078    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:26.656968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:26.656922    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:26.657160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:26.656994    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:27.150331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.150267    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:27.437217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.437127    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:27.575321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.570878    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:27.657416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.657375    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:27.657819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.657444    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:27.738196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:27.738105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:28.657027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:28.656994    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:28.657193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:28.657055    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:29.657253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:29.657208    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 07:18:29.657820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:29.657265    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 07:18:29.695763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:29.695724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:29.844011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:29.843970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:30.616396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:30.616362    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:18:30.662224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:30.662181    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:18:30.671479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:30.671453    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:18:31.649391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:31.649344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:33.699607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:33.699574    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:33.835188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:33.835139    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:33.965573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:33.965481    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:36.106119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:36.106082    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:37.521522 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:18:37.521478    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:18:38.730030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:38.729992    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:40.259814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:40.259782    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr_b823a536-42e3-41fc-bf1c-ee35d171efdd/payload-processing/0.log"
Oct 30 07:18:40.472464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:40.472435    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr_b823a536-42e3-41fc-bf1c-ee35d171efdd/payload-processing/0.log"
Oct 30 07:18:41.161063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.161026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:41.220133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.220065    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:18:41.220913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.220872    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:18:41.333966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.333927    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"]
Oct 30 07:18:41.337564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.337546    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.357561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.357531    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"]
Oct 30 07:18:41.364268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.364232    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:18:41.364471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.364351    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:18:41.364549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.364489    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:18:41.364721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.364702    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:18:41.471135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.471038    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.471313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.471178    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbprq\" (UniqueName: \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.471313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.471216    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.572307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.572254    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.572508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.572366    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbprq\" (UniqueName: \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.572508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.572399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.573033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.573011    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.574766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.574742    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.596113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.596077    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbprq\" (UniqueName: \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\") pod \"maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.655797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.655760    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:41.968440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:41.968410    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"]
Oct 30 07:18:41.969560 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:41.969535    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf51857e_afd3_44ed_a1ed_25b7518d1ced.slice/crio-d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd WatchSource:0}: Error finding container d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd: Status 404 returned error can't find the container with id d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd
Oct 30 07:18:42.779247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:42.779206    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2" event={"ID":"cf51857e-afd3-44ed-a1ed-25b7518d1ced","Type":"ContainerStarted","Data":"4358e9d3ff842b48dd50c44ded73b99a5ab9086b1ade102c9733b5bb5552c629"}
Oct 30 07:18:42.780622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:42.779269    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2" event={"ID":"cf51857e-afd3-44ed-a1ed-25b7518d1ced","Type":"ContainerStarted","Data":"d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd"}
Oct 30 07:18:42.823502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:42.823446    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2" podStartSLOduration=1.823431978 podStartE2EDuration="1.823431978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:42.82269118 +0000 UTC m=+1283.255455090" watchObservedRunningTime="2026-10-30 07:18:42.823431978 +0000 UTC m=+1283.256195870"
Oct 30 07:18:43.346134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.346087    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:43.789141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.789108    6738 generic.go:356] "Generic (PLEG): container finished" podID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerID="115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3" exitCode=0
Oct 30 07:18:43.789629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.789181    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerDied","Data":"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"}
Oct 30 07:18:43.789705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.789682    6738 scope.go:122] "RemoveContainer" containerID="115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"
Oct 30 07:18:43.791031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.791012    6738 generic.go:356] "Generic (PLEG): container finished" podID="cf51857e-afd3-44ed-a1ed-25b7518d1ced" containerID="4358e9d3ff842b48dd50c44ded73b99a5ab9086b1ade102c9733b5bb5552c629" exitCode=0
Oct 30 07:18:43.791131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.791098    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2" event={"ID":"cf51857e-afd3-44ed-a1ed-25b7518d1ced","Type":"ContainerDied","Data":"4358e9d3ff842b48dd50c44ded73b99a5ab9086b1ade102c9733b5bb5552c629"}
Oct 30 07:18:43.791438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.791408    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:43.791506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.791492    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:43.791546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:43.791528    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:18:44.803433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:44.803389    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerStarted","Data":"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"}
Oct 30 07:18:44.937237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:44.937196    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:18:44.937445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:44.937380    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:44.937530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:44.937433    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:45.004774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.004732    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:45.019250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.019220    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\") pod \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") "
Oct 30 07:18:45.019427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.019389    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\" (UniqueName: \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\") pod \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") "
Oct 30 07:18:45.019471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.019445    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\") pod \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\" (UID: \"cf51857e-afd3-44ed-a1ed-25b7518d1ced\") "
Oct 30 07:18:45.019663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.019641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca" pod "cf51857e-afd3-44ed-a1ed-25b7518d1ced" (UID: "cf51857e-afd3-44ed-a1ed-25b7518d1ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:18:45.020043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.020021    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:45.021494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.021468    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca" pod "cf51857e-afd3-44ed-a1ed-25b7518d1ced" (UID: "cf51857e-afd3-44ed-a1ed-25b7518d1ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:18:45.021617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.021565    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq" pod "cf51857e-afd3-44ed-a1ed-25b7518d1ced" (UID: "cf51857e-afd3-44ed-a1ed-25b7518d1ced"). InnerVolumeSpecName "kube-api-access-lbprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:45.121354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.121238    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf51857e-afd3-44ed-a1ed-25b7518d1ced-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:45.121354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.121274    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbprq\" (UniqueName: \"kubernetes.io/projected/cf51857e-afd3-44ed-a1ed-25b7518d1ced-kube-api-access-lbprq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:45.606854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.606791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:45.811856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.811827    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"
Oct 30 07:18:45.812324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.811856    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2" event={"ID":"cf51857e-afd3-44ed-a1ed-25b7518d1ced","Type":"ContainerDied","Data":"d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd"}
Oct 30 07:18:45.812324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.811918    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86143b9b76cfd1c5403d630310eaab72ad74c44c4982967f8a40c360ac7c1bd"
Oct 30 07:18:45.936939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.936900    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:45.937159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:45.936984    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:46.937082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:46.937049    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:46.937537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:46.937105    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:47.666914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:47.666871    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:47.936680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:47.936591    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:47.936680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:47.936647    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:48.034249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:48.034198    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"]
Oct 30 07:18:48.037386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:48.037359    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83fbb16e-7r4qh"]
Oct 30 07:18:48.937348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:48.937261    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:48.937574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:48.937382    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:49.395317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.395265    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"]
Oct 30 07:18:49.396858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.396826    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cf51857e-afd3-44ed-a1ed-25b7518d1ced" containerName="revoke-keys"
Oct 30 07:18:49.396858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.396851    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf51857e-afd3-44ed-a1ed-25b7518d1ced" containerName="revoke-keys"
Oct 30 07:18:49.397030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.396957    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf51857e-afd3-44ed-a1ed-25b7518d1ced" containerName="revoke-keys"
Oct 30 07:18:49.399507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.399489    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.408549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.408514    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.408769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.408734    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.408873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.408791    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nsr\" (UniqueName: \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.413698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.413652    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"]
Oct 30 07:18:49.421794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.421765    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:18:49.421989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.421818    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:18:49.422101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.421928    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:18:49.422195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.421985    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:18:49.509744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.509710    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.509744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.509748    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nsr\" (UniqueName: \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.510041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.509836    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.510551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.510530    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.512193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.512170    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.533364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.533335    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nsr\" (UniqueName: \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\") pod \"maas-api-revoke-sub-f6cfdfb8-cgmhm\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.548109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.548061    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:49.714683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.714592    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:49.717789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.717748    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3755e299-809e-4374-b35e-7a6bc5a7c032" path="/var/lib/kubelet/pods/3755e299-809e-4374-b35e-7a6bc5a7c032/volumes"
Oct 30 07:18:49.936940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.936905    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 07:18:49.937115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:49.936959    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 07:18:50.031652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.031623    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"]
Oct 30 07:18:50.032616 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:50.032588    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e02cca8_5dda_4d57_8a75_c95badd2dab9.slice/crio-2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e WatchSource:0}: Error finding container 2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e: Status 404 returned error can't find the container with id 2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e
Oct 30 07:18:50.856516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.856481    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm" event={"ID":"0e02cca8-5dda-4d57-8a75-c95badd2dab9","Type":"ContainerStarted","Data":"790756428d8ad16ad6b47e18a177fc81ff3a52be1269ac82f2c9e21f380c18bb"}
Oct 30 07:18:50.857041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.856537    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm" event={"ID":"0e02cca8-5dda-4d57-8a75-c95badd2dab9","Type":"ContainerStarted","Data":"2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e"}
Oct 30 07:18:50.898905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.898834    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm" podStartSLOduration=1.898814986 podStartE2EDuration="1.898814986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:50.897463893 +0000 UTC m=+1291.330227787" watchObservedRunningTime="2026-10-30 07:18:50.898814986 +0000 UTC m=+1291.331578880"
Oct 30 07:18:50.941259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.941227    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:18:50.941584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.941562    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:18:50.942414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:50.942395    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:18:51.653758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:51.653717    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:51.867142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:51.867100    6738 generic.go:356] "Generic (PLEG): container finished" podID="0e02cca8-5dda-4d57-8a75-c95badd2dab9" containerID="790756428d8ad16ad6b47e18a177fc81ff3a52be1269ac82f2c9e21f380c18bb" exitCode=0
Oct 30 07:18:51.867537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:51.867162    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm" event={"ID":"0e02cca8-5dda-4d57-8a75-c95badd2dab9","Type":"ContainerDied","Data":"790756428d8ad16ad6b47e18a177fc81ff3a52be1269ac82f2c9e21f380c18bb"}
Oct 30 07:18:53.064924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.064891    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:53.120961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.120923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:53.170282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.170193    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\" (UniqueName: \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\") pod \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") "
Oct 30 07:18:53.170526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.170396    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\") pod \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") "
Oct 30 07:18:53.170526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.170498    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\") pod \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\" (UID: \"0e02cca8-5dda-4d57-8a75-c95badd2dab9\") "
Oct 30 07:18:53.170865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.170836    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca" pod "0e02cca8-5dda-4d57-8a75-c95badd2dab9" (UID: "0e02cca8-5dda-4d57-8a75-c95badd2dab9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:18:53.170991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.170973    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:53.172559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.172528    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca" pod "0e02cca8-5dda-4d57-8a75-c95badd2dab9" (UID: "0e02cca8-5dda-4d57-8a75-c95badd2dab9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:18:53.172654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.172566    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr" pod "0e02cca8-5dda-4d57-8a75-c95badd2dab9" (UID: "0e02cca8-5dda-4d57-8a75-c95badd2dab9"). InnerVolumeSpecName "kube-api-access-z9nsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:53.298982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.273686    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nsr\" (UniqueName: \"kubernetes.io/projected/0e02cca8-5dda-4d57-8a75-c95badd2dab9-kube-api-access-z9nsr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:53.298982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.298874    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02cca8-5dda-4d57-8a75-c95badd2dab9-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:18:53.328397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.310471    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:53.598418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.593354    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:18:53.610317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.609498    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0e02cca8-5dda-4d57-8a75-c95badd2dab9" containerName="revoke-keys"
Oct 30 07:18:53.610317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.609531    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e02cca8-5dda-4d57-8a75-c95badd2dab9" containerName="revoke-keys"
Oct 30 07:18:53.610317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.609779    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e02cca8-5dda-4d57-8a75-c95badd2dab9" containerName="revoke-keys"
Oct 30 07:18:53.614317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.613037    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.636398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.630630    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c1f203-dockercfg-zm96b\""
Oct 30 07:18:53.636398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.630801    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c1f203\""
Oct 30 07:18:53.665691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.665658    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:18:53.747319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.732355    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.747319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.732526    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvr4\" (UniqueName: \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.762589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.762556    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:18:53.775719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.775685    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:18:53.775914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.775819    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.834574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.834524    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvr4\" (UniqueName: \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.834574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.834586    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4d8m\" (UniqueName: \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.834863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.834829    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.834915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.834875    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.835590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.835574    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.858518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.858434    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvr4\" (UniqueName: \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:53.883278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.883243    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm" event={"ID":"0e02cca8-5dda-4d57-8a75-c95badd2dab9","Type":"ContainerDied","Data":"2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e"}
Oct 30 07:18:53.883278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.883271    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"
Oct 30 07:18:53.883553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.883328    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eae6e2cee269a39e3766c37e402816d7450914acd9e5df3e288b9c84aa0573e"
Oct 30 07:18:53.935969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.935932    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.936198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.936082    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4d8m\" (UniqueName: \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.937124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.937097    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.960363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.960335    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4d8m\" (UniqueName: \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\") pod \"payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:53.983269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:53.983234    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:54.093326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.093260    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:54.289286 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:54.289250    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70599fb1_d59f_4905_8bae_8cee724f92af.slice/crio-b4170c87d9859176b3cad78f886202172cc49c1d85328859139899a94408eda8 WatchSource:0}: Error finding container b4170c87d9859176b3cad78f886202172cc49c1d85328859139899a94408eda8: Status 404 returned error can't find the container with id b4170c87d9859176b3cad78f886202172cc49c1d85328859139899a94408eda8
Oct 30 07:18:54.289441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.289314    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:18:54.408643 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:18:54.408610    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ead4a7_92a2_4441_88a3_fc5bc27f0a39.slice/crio-781edf65da7a71610ef2c85964e63f91de171f0248bb9a81ff52f617f13795b3 WatchSource:0}: Error finding container 781edf65da7a71610ef2c85964e63f91de171f0248bb9a81ff52f617f13795b3: Status 404 returned error can't find the container with id 781edf65da7a71610ef2c85964e63f91de171f0248bb9a81ff52f617f13795b3
Oct 30 07:18:54.408921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.408899    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:18:54.465614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.465573    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:54.892763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.892722    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" event={"ID":"70599fb1-d59f-4905-8bae-8cee724f92af","Type":"ContainerStarted","Data":"34dc5e69ba4bae40c8214d907132715591cf6967eb86b65da0babd1771abc777"}
Oct 30 07:18:54.892999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.892783    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" event={"ID":"70599fb1-d59f-4905-8bae-8cee724f92af","Type":"ContainerStarted","Data":"b4170c87d9859176b3cad78f886202172cc49c1d85328859139899a94408eda8"}
Oct 30 07:18:54.892999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.892843    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:18:54.894761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.894732    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" event={"ID":"97ead4a7-92a2-4441-88a3-fc5bc27f0a39","Type":"ContainerStarted","Data":"a4ef97e9fc64c58949718bd4a758d3c6bdbe601aa4d89b45f93cda4044e443b2"}
Oct 30 07:18:54.894890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.894782    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" event={"ID":"97ead4a7-92a2-4441-88a3-fc5bc27f0a39","Type":"ContainerStarted","Data":"781edf65da7a71610ef2c85964e63f91de171f0248bb9a81ff52f617f13795b3"}
Oct 30 07:18:54.894890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.894813    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:18:54.937472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:54.937411    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" podStartSLOduration=1.937395462 podStartE2EDuration="1.937395462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:54.933979308 +0000 UTC m=+1295.366743203" watchObservedRunningTime="2026-10-30 07:18:54.937395462 +0000 UTC m=+1295.370159355"
Oct 30 07:18:55.005071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:55.005010    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" podStartSLOduration=2.004994232 podStartE2EDuration="2.004994232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:55.001401605 +0000 UTC m=+1295.434165497" watchObservedRunningTime="2026-10-30 07:18:55.004994232 +0000 UTC m=+1295.437758184"
Oct 30 07:18:57.065072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:57.065032    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:18:57.102147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:57.102114    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:18:57.223025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:57.222773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:57.353746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:57.350603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:57.464223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:57.464188    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:18:59.795544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:18:59.795493    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:00.138045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.138007    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:00.501100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.501008    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:19:00.501268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.501240    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy" containerID="cri-o://a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07" gracePeriod=30
Oct 30 07:19:00.513323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.513278    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:19:00.513531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.513505    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy" containerID="cri-o://e80bd7a42c3d1cc23ca09872afc6e9dd94ea90db8aed1245a6282131d0d5c21d" gracePeriod=30
Oct 30 07:19:00.517121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.517088    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:19:00.517419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.517381    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy" containerID="cri-o://96fb11b269c6d267e1499d787cda5a547ab2660bf51584e3d80ed982cf1cfc29" gracePeriod=30
Oct 30 07:19:00.526024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.525996    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:19:00.526189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:00.526171    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy" containerID="cri-o://57eb82b5866a8543204cc24e201867ab74eb6d1532a7225c36c3734c1a224163" gracePeriod=30
Oct 30 07:19:02.030575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.030537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:02.068391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.067395    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" podUID="e6937871-1215-4f8a-a1c2-3886b26f7021" containerName="payload-processing" containerID="cri-o://96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281" gracePeriod=25
Oct 30 07:19:02.106147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.105457    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" podUID="744bc65f-efcd-4868-a43a-904c4cadb8aa" containerName="payload-pre-processing" containerID="cri-o://4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7" gracePeriod=25
Oct 30 07:19:02.449791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.449758    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:19:02.452324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.452257    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\" (UniqueName: \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\") pod \"e6937871-1215-4f8a-a1c2-3886b26f7021\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") "
Oct 30 07:19:02.453008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.452593    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\") pod \"e6937871-1215-4f8a-a1c2-3886b26f7021\" (UID: \"e6937871-1215-4f8a-a1c2-3886b26f7021\") "
Oct 30 07:19:02.453008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.452963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume" pod "e6937871-1215-4f8a-a1c2-3886b26f7021" (UID: "e6937871-1215-4f8a-a1c2-3886b26f7021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:02.453333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.453277    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6937871-1215-4f8a-a1c2-3886b26f7021-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:02.455604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.455580    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh" pod "e6937871-1215-4f8a-a1c2-3886b26f7021" (UID: "e6937871-1215-4f8a-a1c2-3886b26f7021"). InnerVolumeSpecName "kube-api-access-8t4qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:02.474731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.474697    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:19:02.554346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.554253    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\") pod \"744bc65f-efcd-4868-a43a-904c4cadb8aa\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") "
Oct 30 07:19:02.554511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.554469    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\" (UniqueName: \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\") pod \"744bc65f-efcd-4868-a43a-904c4cadb8aa\" (UID: \"744bc65f-efcd-4868-a43a-904c4cadb8aa\") "
Oct 30 07:19:02.554668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.554646    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume" pod "744bc65f-efcd-4868-a43a-904c4cadb8aa" (UID: "744bc65f-efcd-4868-a43a-904c4cadb8aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:02.556602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.556347    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4qh\" (UniqueName: \"kubernetes.io/projected/e6937871-1215-4f8a-a1c2-3886b26f7021-kube-api-access-8t4qh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:02.556602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.556391    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744bc65f-efcd-4868-a43a-904c4cadb8aa-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:02.561312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.559743    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf" pod "744bc65f-efcd-4868-a43a-904c4cadb8aa" (UID: "744bc65f-efcd-4868-a43a-904c4cadb8aa"). InnerVolumeSpecName "kube-api-access-k2ngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:02.657930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.657885    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ngf\" (UniqueName: \"kubernetes.io/projected/744bc65f-efcd-4868-a43a-904c4cadb8aa-kube-api-access-k2ngf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:02.975407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.975269    6738 generic.go:356] "Generic (PLEG): container finished" podID="744bc65f-efcd-4868-a43a-904c4cadb8aa" containerID="4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7" exitCode=0
Oct 30 07:19:02.975407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.975329    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" event={"ID":"744bc65f-efcd-4868-a43a-904c4cadb8aa","Type":"ContainerDied","Data":"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"}
Oct 30 07:19:02.975407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.975362    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"
Oct 30 07:19:02.975407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.975403    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv" event={"ID":"744bc65f-efcd-4868-a43a-904c4cadb8aa","Type":"ContainerDied","Data":"19bbe58d08bb2304918f1bd67ea7f0f779707999674354d064aed1ced58efe4b"}
Oct 30 07:19:02.975758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.975437    6738 scope.go:122] "RemoveContainer" containerID="4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"
Oct 30 07:19:02.977130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.977108    6738 generic.go:356] "Generic (PLEG): container finished" podID="e6937871-1215-4f8a-a1c2-3886b26f7021" containerID="96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281" exitCode=0
Oct 30 07:19:02.977258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.977140    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" event={"ID":"e6937871-1215-4f8a-a1c2-3886b26f7021","Type":"ContainerDied","Data":"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"}
Oct 30 07:19:02.977258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.977175    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"
Oct 30 07:19:02.977258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.977191    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr" event={"ID":"e6937871-1215-4f8a-a1c2-3886b26f7021","Type":"ContainerDied","Data":"f2fe49c2eb90ed515dee94771930b9467ea48e4787640e883b4ef257fd0156a2"}
Oct 30 07:19:02.990451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.990431    6738 scope.go:122] "RemoveContainer" containerID="4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"
Oct 30 07:19:02.990809 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:19:02.990790    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7\": container with ID starting with 4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7 not found: ID does not exist" containerID="4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"
Oct 30 07:19:02.990898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.990816    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7"} err="failed to get container status \"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7\": rpc error: code = NotFound desc = could not find container \"4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7\": container with ID starting with 4d0a26948dbfc1a2b934fd7f5455c04b63ea6e187857739091b2204c9c4371c7 not found: ID does not exist"
Oct 30 07:19:02.990898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:02.990833    6738 scope.go:122] "RemoveContainer" containerID="96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"
Oct 30 07:19:03.002117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.002084    6738 scope.go:122] "RemoveContainer" containerID="96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"
Oct 30 07:19:03.002499 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:19:03.002460    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281\": container with ID starting with 96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281 not found: ID does not exist" containerID="96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"
Oct 30 07:19:03.002595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.002501    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281"} err="failed to get container status \"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281\": rpc error: code = NotFound desc = could not find container \"96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281\": container with ID starting with 96b208dffc5b2007570282f521ca0335f094fd8f38e93dcc0a5ed76fbcd81281 not found: ID does not exist"
Oct 30 07:19:03.028662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.028623    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:19:03.040875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.040842    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-6ccb6474c4-hkdhr"]
Oct 30 07:19:03.084577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.084546    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:19:03.087589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.087564    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-67f5fb7fff-kngfv"]
Oct 30 07:19:03.693219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.693183    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744bc65f-efcd-4868-a43a-904c4cadb8aa" path="/var/lib/kubelet/pods/744bc65f-efcd-4868-a43a-904c4cadb8aa/volumes"
Oct 30 07:19:03.693693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.693678    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6937871-1215-4f8a-a1c2-3886b26f7021" path="/var/lib/kubelet/pods/e6937871-1215-4f8a-a1c2-3886b26f7021/volumes"
Oct 30 07:19:03.903320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:03.896785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:05.822903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.822811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:05.904133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.894794    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:19:05.911436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.911372    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:19:05.913279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.912837    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:19:05.945318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.920850    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.945318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.920922    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.945318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.920970    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.945318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.921017    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.945318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.921068    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.951883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.951285    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:05.951883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.951871    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:05.953469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.953402    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:05.958049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.957985    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.958953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.958926    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.959738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.959717    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.960076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.960057    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\") pod \"7e34096d-9005-409a-b595-010c2ea73a79\" (UID: \"7e34096d-9005-409a-b595-010c2ea73a79\") "
Oct 30 07:19:05.960922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.960889    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e34096d-9005-409a-b595-010c2ea73a79-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:05.961039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.961027    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:05.961123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.961110    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:05.962935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.951085    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:05.963090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.962337    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:05.979137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.977040    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:05.982376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.980370    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:05.988179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.988148    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "kube-api-access-kjldv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:05.989084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:05.989057    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo" pod "7e34096d-9005-409a-b595-010c2ea73a79" (UID: "7e34096d-9005-409a-b595-010c2ea73a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:06.012478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.012285    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:06.027649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.026270    6738 generic.go:356] "Generic (PLEG): container finished" podID="7e34096d-9005-409a-b595-010c2ea73a79" containerID="a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07" exitCode=0
Oct 30 07:19:06.027649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.026413    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" event={"ID":"7e34096d-9005-409a-b595-010c2ea73a79","Type":"ContainerDied","Data":"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"}
Oct 30 07:19:06.027649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.026469    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p" event={"ID":"7e34096d-9005-409a-b595-010c2ea73a79","Type":"ContainerDied","Data":"ae47d7f5e3e16ccc44138b82a0534b642b5113b5c63643618203b2d1c01de950"}
Oct 30 07:19:06.027649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.026501    6738 scope.go:122] "RemoveContainer" containerID="a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"
Oct 30 07:19:06.027649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.026628    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.038327    6738 generic.go:356] "Generic (PLEG): container finished" podID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerID="57eb82b5866a8543204cc24e201867ab74eb6d1532a7225c36c3734c1a224163" exitCode=0
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.038480    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerDied","Data":"57eb82b5866a8543204cc24e201867ab74eb6d1532a7225c36c3734c1a224163"}
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.040233    6738 generic.go:356] "Generic (PLEG): container finished" podID="21e07602-bf26-4e36-a477-eb9c9555931a" containerID="96fb11b269c6d267e1499d787cda5a547ab2660bf51584e3d80ed982cf1cfc29" exitCode=0
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.040426    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" event={"ID":"21e07602-bf26-4e36-a477-eb9c9555931a","Type":"ContainerDied","Data":"96fb11b269c6d267e1499d787cda5a547ab2660bf51584e3d80ed982cf1cfc29"}
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.042061    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.042218    6738 generic.go:356] "Generic (PLEG): container finished" podID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerID="e80bd7a42c3d1cc23ca09872afc6e9dd94ea90db8aed1245a6282131d0d5c21d" exitCode=0
Oct 30 07:19:06.042962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.042403    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" event={"ID":"e14df479-780b-4d3a-8844-e2f1449c75ed","Type":"ContainerDied","Data":"e80bd7a42c3d1cc23ca09872afc6e9dd94ea90db8aed1245a6282131d0d5c21d"}
Oct 30 07:19:06.055619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.055593    6738 scope.go:122] "RemoveContainer" containerID="a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"
Oct 30 07:19:06.056251 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:19:06.056123    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07\": container with ID starting with a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07 not found: ID does not exist" containerID="a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"
Oct 30 07:19:06.056251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.056164    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07"} err="failed to get container status \"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07\": rpc error: code = NotFound desc = could not find container \"a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07\": container with ID starting with a85e1fbb6a6794284985edcbf973ff76aaa05272234f6ea80e06427d3828ad07 not found: ID does not exist"
Oct 30 07:19:06.056251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.056186    6738 scope.go:122] "RemoveContainer" containerID="1f936c455a6871cf6b997e8679e50bfa95febda20c6a79445b05a872fb1ba7d7"
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063648    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjldv\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-kube-api-access-kjldv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063680    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e34096d-9005-409a-b595-010c2ea73a79-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063702    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063717    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063734    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e34096d-9005-409a-b595-010c2ea73a79-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.065260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.063749    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e34096d-9005-409a-b595-010c2ea73a79-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.089090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.089012    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:19:06.092513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.092484    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:19:06.118516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.118481    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:19:06.126523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.126492    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-t8v8p"]
Oct 30 07:19:06.164213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164179    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.164399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164241    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.164399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164281    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.164399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164388    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.164586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164434    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.164586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164464    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.164586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164493    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.164586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164531    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.164586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164569    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.164841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164598    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.164841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164633    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.164841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164689    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.164841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164755    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164869    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164866    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164903    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164929    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.164968    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165036    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165069    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165107    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165165    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165197    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165253    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165315    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165289    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:06.165366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165359    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.165761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165437    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\") pod \"f94efc78-0fa1-454c-b59a-e65491208bfc\" (UID: \"f94efc78-0fa1-454c-b59a-e65491208bfc\") "
Oct 30 07:19:06.165761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165480    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\") pod \"21e07602-bf26-4e36-a477-eb9c9555931a\" (UID: \"21e07602-bf26-4e36-a477-eb9c9555931a\") "
Oct 30 07:19:06.165761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165511    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\") pod \"e14df479-780b-4d3a-8844-e2f1449c75ed\" (UID: \"e14df479-780b-4d3a-8844-e2f1449c75ed\") "
Oct 30 07:19:06.165761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165611    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.165957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.165799    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.166325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166257    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.166435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166332    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.166435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166352    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94efc78-0fa1-454c-b59a-e65491208bfc-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.166435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166368    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.166435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166380    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.166662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166597    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.166662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.166598    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.167895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.167865    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.167903    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.168116    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.168412    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.168889    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.168990    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.170708    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.170857    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.171633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171603    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171730    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "kube-api-access-7kc5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171750    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171792    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171841    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.171847    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.172393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.172015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.174414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.174165    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.178485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.178460    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo" pod "e14df479-780b-4d3a-8844-e2f1449c75ed" (UID: "e14df479-780b-4d3a-8844-e2f1449c75ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:06.178485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.178467    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:06.178623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.178498    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62" pod "f94efc78-0fa1-454c-b59a-e65491208bfc" (UID: "f94efc78-0fa1-454c-b59a-e65491208bfc"). InnerVolumeSpecName "kube-api-access-w7s62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.178623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.178530    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t" pod "21e07602-bf26-4e36-a477-eb9c9555931a" (UID: "21e07602-bf26-4e36-a477-eb9c9555931a"). InnerVolumeSpecName "kube-api-access-nl46t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:06.268537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268493    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268539    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268554    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268569    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268585    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268602    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14df479-780b-4d3a-8844-e2f1449c75ed-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268615    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268628    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268641    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e07602-bf26-4e36-a477-eb9c9555931a-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268654    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7s62\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-kube-api-access-w7s62\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268668    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94efc78-0fa1-454c-b59a-e65491208bfc-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268682    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e07602-bf26-4e36-a477-eb9c9555931a-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268697    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268710    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268722    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268735    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl46t\" (UniqueName: \"kubernetes.io/projected/21e07602-bf26-4e36-a477-eb9c9555931a-kube-api-access-nl46t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268758    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268773    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kc5w\" (UniqueName: \"kubernetes.io/projected/e14df479-780b-4d3a-8844-e2f1449c75ed-kube-api-access-7kc5w\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268787    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268804    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94efc78-0fa1-454c-b59a-e65491208bfc-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268832    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14df479-780b-4d3a-8844-e2f1449c75ed-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.268835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.268846    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e07602-bf26-4e36-a477-eb9c9555931a-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:06.386493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.386454    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:19:06.386682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:06.386548    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:19:07.052634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.052602    6738 generic.go:356] "Generic (PLEG): container finished" podID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerID="ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1" exitCode=0
Oct 30 07:19:07.053114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.052675    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerDied","Data":"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"}
Oct 30 07:19:07.053233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.053216    6738 scope.go:122] "RemoveContainer" containerID="ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"
Oct 30 07:19:07.054223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.054129    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"
Oct 30 07:19:07.054223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.054153    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf" event={"ID":"f94efc78-0fa1-454c-b59a-e65491208bfc","Type":"ContainerDied","Data":"ed19b9ea98e556cfc01bcc180b6b36633508da857644ed4fb08ba4733364f2db"}
Oct 30 07:19:07.054223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.054197    6738 scope.go:122] "RemoveContainer" containerID="57eb82b5866a8543204cc24e201867ab74eb6d1532a7225c36c3734c1a224163"
Oct 30 07:19:07.055062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.055027    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:07.055149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.055125    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:07.055304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.055174    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:07.055797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.055767    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"
Oct 30 07:19:07.055866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.055819    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74" event={"ID":"21e07602-bf26-4e36-a477-eb9c9555931a","Type":"ContainerDied","Data":"84c1c7fb936a1f09100e24e712529d7823f396c3b9759cf49ff75861013c71d5"}
Oct 30 07:19:07.061334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.057622    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"
Oct 30 07:19:07.061334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.057671    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt" event={"ID":"e14df479-780b-4d3a-8844-e2f1449c75ed","Type":"ContainerDied","Data":"b99614409e0a9589a94933df6c1a32911ad5682c7062a9e58adb7d9be9eb7f0e"}
Oct 30 07:19:07.070695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.070656    6738 scope.go:122] "RemoveContainer" containerID="96fb11b269c6d267e1499d787cda5a547ab2660bf51584e3d80ed982cf1cfc29"
Oct 30 07:19:07.084727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.084695    6738 scope.go:122] "RemoveContainer" containerID="e80bd7a42c3d1cc23ca09872afc6e9dd94ea90db8aed1245a6282131d0d5c21d"
Oct 30 07:19:07.169481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.169446    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:19:07.172529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.172504    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-zh7lt"]
Oct 30 07:19:07.227342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.227308    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:19:07.232566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.232539    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vqfvf"]
Oct 30 07:19:07.260797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.260759    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:19:07.265867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.265833    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hvn74"]
Oct 30 07:19:07.520063 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:19:07.520024    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:19:07.691705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.691670    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" path="/var/lib/kubelet/pods/21e07602-bf26-4e36-a477-eb9c9555931a/volumes"
Oct 30 07:19:07.692211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.692189    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e34096d-9005-409a-b595-010c2ea73a79" path="/var/lib/kubelet/pods/7e34096d-9005-409a-b595-010c2ea73a79/volumes"
Oct 30 07:19:07.692707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.692685    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" path="/var/lib/kubelet/pods/e14df479-780b-4d3a-8844-e2f1449c75ed/volumes"
Oct 30 07:19:07.693522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:07.693502    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" path="/var/lib/kubelet/pods/f94efc78-0fa1-454c-b59a-e65491208bfc/volumes"
Oct 30 07:19:08.088318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.067327    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerStarted","Data":"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"}
Oct 30 07:19:08.107182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.107149    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:19:08.107413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.107358    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:08.107534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.107410    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:08.217156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.217118    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:08.250209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250165    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"]
Oct 30 07:19:08.250833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250814    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:08.250833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250831    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250847    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250852    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250876    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250881    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250903    6738 cpu_manager.go:414] "containerMap: removing container" podUID="744bc65f-efcd-4868-a43a-904c4cadb8aa" containerName="payload-pre-processing"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250908    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744bc65f-efcd-4868-a43a-904c4cadb8aa" containerName="payload-pre-processing"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250916    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e6937871-1215-4f8a-a1c2-3886b26f7021" containerName="payload-processing"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250921    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6937871-1215-4f8a-a1c2-3886b26f7021" containerName="payload-processing"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250931    6738 cpu_manager.go:414] "containerMap: removing container" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.250936    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy"
Oct 30 07:19:08.251013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251018    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:08.251592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251028    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e34096d-9005-409a-b595-010c2ea73a79" containerName="istio-proxy"
Oct 30 07:19:08.251592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251037    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14df479-780b-4d3a-8844-e2f1449c75ed" containerName="istio-proxy"
Oct 30 07:19:08.251592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251048    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="744bc65f-efcd-4868-a43a-904c4cadb8aa" containerName="payload-pre-processing"
Oct 30 07:19:08.251592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251054    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6937871-1215-4f8a-a1c2-3886b26f7021" containerName="payload-processing"
Oct 30 07:19:08.251592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.251061    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e07602-bf26-4e36-a477-eb9c9555931a" containerName="istio-proxy"
Oct 30 07:19:08.271317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.265250    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.289812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.288674    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:19:08.289812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.289457    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:19:08.289812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.289548    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:19:08.289812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.289636    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:19:08.298323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.294889    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"]
Oct 30 07:19:08.407918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.407884    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bz84\" (UniqueName: \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.408102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.408021    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.408201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.408180    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.509475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.509434    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bz84\" (UniqueName: \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.509686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.509552    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.509686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.509678    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.510364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.510340    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.512237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.512212    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.533289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.533259    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bz84\" (UniqueName: \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\") pod \"maas-api-revoke-sub-8238b38a-pvczv\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.624906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.624871    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:08.693375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.692716    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:08.958315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:08.958275    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"]
Oct 30 07:19:08.958999 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:08.958968    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc8e5fb_9c51_466b_b4a0_3e681122f467.slice/crio-16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1 WatchSource:0}: Error finding container 16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1: Status 404 returned error can't find the container with id 16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1
Oct 30 07:19:09.106922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:09.106885    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv" event={"ID":"7dc8e5fb-9c51-466b-b4a0-3e681122f467","Type":"ContainerStarted","Data":"16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1"}
Oct 30 07:19:09.107306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:09.107175    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:09.107306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:09.107228    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:09.523710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:09.523540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:10.026149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.026091    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:19:10.059320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.051783    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:10.063320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.059560    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:10.101317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.090600    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94efc78-0fa1-454c-b59a-e65491208bfc" containerName="istio-proxy"
Oct 30 07:19:10.124073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.111903    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.317369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.249750    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391455    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391519    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391614    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391647    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391771    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391817    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdjd\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.391915    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.367006    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2340b1-openshift-default-dockercfg-djkl8\""
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.370472    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:10.417449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.392215    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:10.425347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.425224    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493486    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493548    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493633    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493665    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493697    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493768    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493808    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdjd\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.493892    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.494034    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.494634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.502355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.494922    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.510064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.509752    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.510064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.510021    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.510340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.510093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.548068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.547984    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:19:10.633503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.605699    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.634866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.606938    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.651314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.644397    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:10.678332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.657942    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv" event={"ID":"7dc8e5fb-9c51-466b-b4a0-3e681122f467","Type":"ContainerStarted","Data":"cdd17a97c7789bd1d719f11bc9bbbe83487cd29b61f29bef6667d0a4d71578c1"}
Oct 30 07:19:10.704327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.703889    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdjd\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.728345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.705489    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\") pod \"e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") " pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:10.991627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:10.980874    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"]
Oct 30 07:19:11.025415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.021635    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:11.163508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.162884    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:11.163508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.162943    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:11.245797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.245717    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"]
Oct 30 07:19:11.246118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.246097    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.408146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.399274    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.408146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.399496    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxld\" (UniqueName: \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.408963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.408661    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.545173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.543620    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxld\" (UniqueName: \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.545173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.543680    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.545173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.543903    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.546929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.546776    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.638578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.638533    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.790832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.790749    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxld\" (UniqueName: \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\") pod \"maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:11.797307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.796278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:11.839508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.835015    6738 generic.go:356] "Generic (PLEG): container finished" podID="7dc8e5fb-9c51-466b-b4a0-3e681122f467" containerID="cdd17a97c7789bd1d719f11bc9bbbe83487cd29b61f29bef6667d0a4d71578c1" exitCode=0
Oct 30 07:19:11.839508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.835102    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv" event={"ID":"7dc8e5fb-9c51-466b-b4a0-3e681122f467","Type":"ContainerDied","Data":"cdd17a97c7789bd1d719f11bc9bbbe83487cd29b61f29bef6667d0a4d71578c1"}
Oct 30 07:19:11.890644 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:11.890612    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfdc2bf_8d8c_4396_bf35_43a515a5fd58.slice/crio-769fc8eafa589d8f255ccef4f333475d80f44b8b71d5806e946c2f6575fda3d5 WatchSource:0}: Error finding container 769fc8eafa589d8f255ccef4f333475d80f44b8b71d5806e946c2f6575fda3d5: Status 404 returned error can't find the container with id 769fc8eafa589d8f255ccef4f333475d80f44b8b71d5806e946c2f6575fda3d5
Oct 30 07:19:11.906523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.906472    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:11.906781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.906754    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:11.906994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.906920    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:19:11.929515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.925494    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:19:11.978322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:11.959419    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:12.113398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.112986    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:12.113398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.113051    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:12.544471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.542866    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:12.553580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.549983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\") pod \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") "
Oct 30 07:19:12.553580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.550050    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\" (UniqueName: \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\") pod \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") "
Oct 30 07:19:12.553580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.550147    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\") pod \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\" (UID: \"7dc8e5fb-9c51-466b-b4a0-3e681122f467\") "
Oct 30 07:19:12.553580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.550927    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca" pod "7dc8e5fb-9c51-466b-b4a0-3e681122f467" (UID: "7dc8e5fb-9c51-466b-b4a0-3e681122f467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:12.724094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.671172    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:12.724094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.724054    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84" pod "7dc8e5fb-9c51-466b-b4a0-3e681122f467" (UID: "7dc8e5fb-9c51-466b-b4a0-3e681122f467"). InnerVolumeSpecName "kube-api-access-5bz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:12.727918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.727850    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca" pod "7dc8e5fb-9c51-466b-b4a0-3e681122f467" (UID: "7dc8e5fb-9c51-466b-b4a0-3e681122f467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:19:12.751989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.747177    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"]
Oct 30 07:19:12.809578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.799426    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc8e5fb-9c51-466b-b4a0-3e681122f467-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:12.809578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.806844    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bz84\" (UniqueName: \"kubernetes.io/projected/7dc8e5fb-9c51-466b-b4a0-3e681122f467-kube-api-access-5bz84\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:12.847439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.847410    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"
Oct 30 07:19:12.847590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.847434    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv" event={"ID":"7dc8e5fb-9c51-466b-b4a0-3e681122f467","Type":"ContainerDied","Data":"16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1"}
Oct 30 07:19:12.847590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.847494    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16040e0d4ff3ed7ea3ccc0aa6b625e572e1bc9a697893d1cad8eacd0e8b5c4b1"
Oct 30 07:19:12.849270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.849247    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd" event={"ID":"6722aea5-7541-48f7-864e-2e4e7991e20f","Type":"ContainerStarted","Data":"859986a5a21b53e887ae598ead6cd2b44e1bf6590d1c518e19dda68959a317af"}
Oct 30 07:19:12.852349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.852322    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerStarted","Data":"fab15fbcee5b3b59fef0ff4b9949243cb21c6ebdf5b3c075e8197f774d2ca551"}
Oct 30 07:19:12.852544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.852526    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerStarted","Data":"769fc8eafa589d8f255ccef4f333475d80f44b8b71d5806e946c2f6575fda3d5"}
Oct 30 07:19:12.907957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:12.907884    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podStartSLOduration=3.9078610879999998 podStartE2EDuration="3.907861088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:12.901369712 +0000 UTC m=+1313.334133607" watchObservedRunningTime="2026-10-30 07:19:12.907861088 +0000 UTC m=+1313.340624982"
Oct 30 07:19:13.029679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.029590    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:13.030678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.029946    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:19:13.030916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.030890    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:19:13.127421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.119475    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 07:19:13.127421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.119536    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 07:19:13.861826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.861789    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd" event={"ID":"6722aea5-7541-48f7-864e-2e4e7991e20f","Type":"ContainerStarted","Data":"068b3ff63204f4ff26c060133f7aa408f14bcd43477d63baf6f459b8da440438"}
Oct 30 07:19:13.915388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:13.915309    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd" podStartSLOduration=3.915279215 podStartE2EDuration="3.915279215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:13.912176758 +0000 UTC m=+1314.344940654" watchObservedRunningTime="2026-10-30 07:19:13.915279215 +0000 UTC m=+1314.348043108"
Oct 30 07:19:14.030815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.030785    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:14.151606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.151565    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:14.179888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.179797    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:19:14.180833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.180808    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:19:14.181856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.181830    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:19:14.871308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.871262    6738 generic.go:356] "Generic (PLEG): container finished" podID="6722aea5-7541-48f7-864e-2e4e7991e20f" containerID="068b3ff63204f4ff26c060133f7aa408f14bcd43477d63baf6f459b8da440438" exitCode=0
Oct 30 07:19:14.871777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.871327    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd" event={"ID":"6722aea5-7541-48f7-864e-2e4e7991e20f","Type":"ContainerDied","Data":"068b3ff63204f4ff26c060133f7aa408f14bcd43477d63baf6f459b8da440438"}
Oct 30 07:19:14.871892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.871871    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:14.872787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:14.872769    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:19:15.383952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:15.383903    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:15.530639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:15.530600    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:16.030140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.030114    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:16.155162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.155071    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\" (UniqueName: \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\") pod \"6722aea5-7541-48f7-864e-2e4e7991e20f\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") "
Oct 30 07:19:16.155162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.155134    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\") pod \"6722aea5-7541-48f7-864e-2e4e7991e20f\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") "
Oct 30 07:19:16.155463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.155379    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\") pod \"6722aea5-7541-48f7-864e-2e4e7991e20f\" (UID: \"6722aea5-7541-48f7-864e-2e4e7991e20f\") "
Oct 30 07:19:16.155598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.155574    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca" pod "6722aea5-7541-48f7-864e-2e4e7991e20f" (UID: "6722aea5-7541-48f7-864e-2e4e7991e20f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:16.155887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.155867    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:16.157286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.157265    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld" pod "6722aea5-7541-48f7-864e-2e4e7991e20f" (UID: "6722aea5-7541-48f7-864e-2e4e7991e20f"). InnerVolumeSpecName "kube-api-access-xnxld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:16.157435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.157411    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca" pod "6722aea5-7541-48f7-864e-2e4e7991e20f" (UID: "6722aea5-7541-48f7-864e-2e4e7991e20f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:19:16.241252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.241210    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:16.257220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.257177    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722aea5-7541-48f7-864e-2e4e7991e20f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:16.257220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.257213    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnxld\" (UniqueName: \"kubernetes.io/projected/6722aea5-7541-48f7-864e-2e4e7991e20f-kube-api-access-xnxld\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:16.890007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.889969    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd" event={"ID":"6722aea5-7541-48f7-864e-2e4e7991e20f","Type":"ContainerDied","Data":"859986a5a21b53e887ae598ead6cd2b44e1bf6590d1c518e19dda68959a317af"}
Oct 30 07:19:16.890007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.890001    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"
Oct 30 07:19:16.890239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:16.890021    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859986a5a21b53e887ae598ead6cd2b44e1bf6590d1c518e19dda68959a317af"
Oct 30 07:19:17.115899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.115864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:17.244403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.244280    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:17.296058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296026    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:19:17.296691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296676    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6722aea5-7541-48f7-864e-2e4e7991e20f" containerName="revoke-keys"
Oct 30 07:19:17.296752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296692    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6722aea5-7541-48f7-864e-2e4e7991e20f" containerName="revoke-keys"
Oct 30 07:19:17.296752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296715    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7dc8e5fb-9c51-466b-b4a0-3e681122f467" containerName="revoke-keys"
Oct 30 07:19:17.296752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296721    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc8e5fb-9c51-466b-b4a0-3e681122f467" containerName="revoke-keys"
Oct 30 07:19:17.296865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296855    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6722aea5-7541-48f7-864e-2e4e7991e20f" containerName="revoke-keys"
Oct 30 07:19:17.296898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.296870    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc8e5fb-9c51-466b-b4a0-3e681122f467" containerName="revoke-keys"
Oct 30 07:19:17.299626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.299606    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.316991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.316962    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2340b1\""
Oct 30 07:19:17.317156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.317054    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2340b1-dockercfg-4hlvw\""
Oct 30 07:19:17.321369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.321342    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:19:17.418329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.409286    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.418329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.409393    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf9n\" (UniqueName: \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.520333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.520220    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.528115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.528073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf9n\" (UniqueName: \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.530280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.530248    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.542209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.542171    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:19:17.548111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.548085    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:19:17.549396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.548464    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.577322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.570051    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf9n\" (UniqueName: \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\") pod \"payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.613388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.613351    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:17.630339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.630309    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kww67\" (UniqueName: \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.630628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.630600    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.732190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.732150    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.732421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.732271    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kww67\" (UniqueName: \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.732929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.732898    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.756419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.756384    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kww67\" (UniqueName: \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\") pod \"payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.908221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.908182    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:17.923650 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:17.923617    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7e5d4b_06c5_4b44_af55_352f0667354d.slice/crio-61fc5880ea13bf56085a6570efba01a829a94f217705c3b31a582e8c74e83558 WatchSource:0}: Error finding container 61fc5880ea13bf56085a6570efba01a829a94f217705c3b31a582e8c74e83558: Status 404 returned error can't find the container with id 61fc5880ea13bf56085a6570efba01a829a94f217705c3b31a582e8c74e83558
Oct 30 07:19:17.923867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:17.923841    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:19:18.265819 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:18.265784    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac77b74_ddb1_4bb1_84c2_962304650043.slice/crio-5e22c5612072c7ee4168b471154fd4112b49ded97f835c72d9ebae5f9a55e60d WatchSource:0}: Error finding container 5e22c5612072c7ee4168b471154fd4112b49ded97f835c72d9ebae5f9a55e60d: Status 404 returned error can't find the container with id 5e22c5612072c7ee4168b471154fd4112b49ded97f835c72d9ebae5f9a55e60d
Oct 30 07:19:18.272254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.272228    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:19:18.464148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.464102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:18.909643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.909606    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" event={"ID":"6ac77b74-ddb1-4bb1-84c2-962304650043","Type":"ContainerStarted","Data":"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"}
Oct 30 07:19:18.909851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.909663    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" event={"ID":"6ac77b74-ddb1-4bb1-84c2-962304650043","Type":"ContainerStarted","Data":"5e22c5612072c7ee4168b471154fd4112b49ded97f835c72d9ebae5f9a55e60d"}
Oct 30 07:19:18.909851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.909692    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:18.911307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.911264    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" event={"ID":"6b7e5d4b-06c5-4b44-af55-352f0667354d","Type":"ContainerStarted","Data":"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"}
Oct 30 07:19:18.911453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.911326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" event={"ID":"6b7e5d4b-06c5-4b44-af55-352f0667354d","Type":"ContainerStarted","Data":"61fc5880ea13bf56085a6570efba01a829a94f217705c3b31a582e8c74e83558"}
Oct 30 07:19:18.911453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.911357    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:18.955912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:18.955860    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" podStartSLOduration=1.955847884 podStartE2EDuration="1.955847884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:18.952279084 +0000 UTC m=+1319.385042992" watchObservedRunningTime="2026-10-30 07:19:18.955847884 +0000 UTC m=+1319.388611777"
Oct 30 07:19:19.009402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:19.009343    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" podStartSLOduration=2.009326492 podStartE2EDuration="2.009326492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:19.007489706 +0000 UTC m=+1319.440253603" watchObservedRunningTime="2026-10-30 07:19:19.009326492 +0000 UTC m=+1319.442090384"
Oct 30 07:19:19.041911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:19.041872    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"]
Oct 30 07:19:19.045848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:19.045819    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5bde99-84zb8"]
Oct 30 07:19:19.691631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:19.691597    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86388548-e642-49e1-a78b-8e1c4a2aa39a" path="/var/lib/kubelet/pods/86388548-e642-49e1-a78b-8e1c4a2aa39a/volumes"
Oct 30 07:19:19.892961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:19.892923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:20.135420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.129394    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:20.476383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.474518    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:19:20.488707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.484733    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:19:20.488707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.484865    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.513954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.513918    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f7467d-dockercfg-9b8rx\""
Oct 30 07:19:20.523239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.523212    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f7467d\""
Oct 30 07:19:20.537855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.537815    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbk2\" (UniqueName: \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.538001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.537976    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.622091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.622061    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:19:20.625390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.625372    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.639454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.639420    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8f6\" (UniqueName: \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.639586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.639476    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbk2\" (UniqueName: \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.639683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.639650    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.639858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.639793    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.640934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.640901    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.644636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.644610    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:19:20.663332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.663267    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbk2\" (UniqueName: \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\") pod \"payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.741594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.741503    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.742093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.741680    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8f6\" (UniqueName: \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.742442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.742420    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.765394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.765367    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8f6\" (UniqueName: \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\") pod \"payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:20.805305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.805269    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:20.939886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:20.939842    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:21.121801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.121736    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:19:21.127990 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:21.127959    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcdfd68_aaa5_4187_bdc2_2b7558f81351.slice/crio-22b2a8331cd06945402cf065d5c39d0e214ef2152f6d271bfeccd79564f761a6 WatchSource:0}: Error finding container 22b2a8331cd06945402cf065d5c39d0e214ef2152f6d271bfeccd79564f761a6: Status 404 returned error can't find the container with id 22b2a8331cd06945402cf065d5c39d0e214ef2152f6d271bfeccd79564f761a6
Oct 30 07:19:21.197923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.197885    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:21.254378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.253429    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:19:21.941333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.941271    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" event={"ID":"c33ca108-63a1-4c34-88c9-2d6951722ab2","Type":"ContainerStarted","Data":"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"}
Oct 30 07:19:21.941725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.941353    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" event={"ID":"c33ca108-63a1-4c34-88c9-2d6951722ab2","Type":"ContainerStarted","Data":"ffb0b6c0ace2387c93ebe999bd92a6030a2a2d63b165caff2fabeb0c2d9fbfe8"}
Oct 30 07:19:21.942215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.941382    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:21.943626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.943602    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" event={"ID":"1dcdfd68-aaa5-4187-bdc2-2b7558f81351","Type":"ContainerStarted","Data":"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"}
Oct 30 07:19:21.943731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.943642    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" event={"ID":"1dcdfd68-aaa5-4187-bdc2-2b7558f81351","Type":"ContainerStarted","Data":"22b2a8331cd06945402cf065d5c39d0e214ef2152f6d271bfeccd79564f761a6"}
Oct 30 07:19:21.943731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.943663    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:21.985103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:21.985042    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" podStartSLOduration=1.985021446 podStartE2EDuration="1.985021446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:21.982060283 +0000 UTC m=+1322.414824187" watchObservedRunningTime="2026-10-30 07:19:21.985021446 +0000 UTC m=+1322.417785341"
Oct 30 07:19:22.026406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:22.026343    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" podStartSLOduration=2.026328203 podStartE2EDuration="2.026328203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:22.023529575 +0000 UTC m=+1322.456293469" watchObservedRunningTime="2026-10-30 07:19:22.026328203 +0000 UTC m=+1322.459092095"
Oct 30 07:19:23.141619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.141571    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-124.ec2.internal" podUID="bc1bcd324f5af093ecce999f61f1d18f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:19:23.361059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.361026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:23.474683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.474596    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:19:23.474847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.474798    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerName="payload-processing" containerID="cri-o://ed0eb6445a3ea5284828a1a128374c9795fe4a02fb3c396d502521af1da31eb2" gracePeriod=30
Oct 30 07:19:23.498334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.496135    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:19:23.498334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.496403    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerName="payload-pre-processing" containerID="cri-o://e9751e7568da2f3b45250a145c5a39339593fe56bc7545d0fd21abf41616b183" gracePeriod=30
Oct 30 07:19:23.527978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.526998    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.159:9004: connect: connection refused"
Oct 30 07:19:23.527978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.527028    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.160:9004: connect: connection refused"
Oct 30 07:19:23.561853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.561827    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:23.679411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.678740    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:23.712078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.712038    6738 scope.go:122] "RemoveContainer" containerID="8858d0e1809ac2a262755601b24ca5a1583648c5f168daf1d7dff4d63d373952"
Oct 30 07:19:23.727430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:23.727401    6738 scope.go:122] "RemoveContainer" containerID="4969041ac7d2640d3bf671f3043b473cb0ce805fb50167a69bd0d7512798d5c1"
Oct 30 07:19:24.029476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.029445    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:19:24.031916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.031891    6738 generic.go:356] "Generic (PLEG): container finished" podID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerID="ed0eb6445a3ea5284828a1a128374c9795fe4a02fb3c396d502521af1da31eb2" exitCode=0
Oct 30 07:19:24.032008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.031996    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" event={"ID":"0ac45e3e-179c-4aa8-8a71-5bf626280bbc","Type":"ContainerDied","Data":"ed0eb6445a3ea5284828a1a128374c9795fe4a02fb3c396d502521af1da31eb2"}
Oct 30 07:19:24.032066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.032034    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl" event={"ID":"0ac45e3e-179c-4aa8-8a71-5bf626280bbc","Type":"ContainerDied","Data":"6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f"}
Oct 30 07:19:24.032066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.032050    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6214be174abad855fd58a900189337e254dd32e970afecbf6ff2bbe91dab012f"
Oct 30 07:19:24.032240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.032219    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:19:24.048936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.048795    6738 generic.go:356] "Generic (PLEG): container finished" podID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerID="e9751e7568da2f3b45250a145c5a39339593fe56bc7545d0fd21abf41616b183" exitCode=0
Oct 30 07:19:24.048936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.048894    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h" event={"ID":"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e","Type":"ContainerDied","Data":"e9751e7568da2f3b45250a145c5a39339593fe56bc7545d0fd21abf41616b183"}
Oct 30 07:19:24.049058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.048954    6738 scope.go:122] "RemoveContainer" containerID="e9751e7568da2f3b45250a145c5a39339593fe56bc7545d0fd21abf41616b183"
Oct 30 07:19:24.049127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.049109    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"
Oct 30 07:19:24.117876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.117839    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:24.127075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127030    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\" (UniqueName: \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\") pod \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") "
Oct 30 07:19:24.127285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127118    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\") pod \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") "
Oct 30 07:19:24.127285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127281    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\") pod \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\" (UID: \"f1b1afb3-5ea7-43f9-9623-5b08b8e0340e\") "
Oct 30 07:19:24.127546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127510    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\" (UniqueName: \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\") pod \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\" (UID: \"0ac45e3e-179c-4aa8-8a71-5bf626280bbc\") "
Oct 30 07:19:24.127697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127672    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume" pod "0ac45e3e-179c-4aa8-8a71-5bf626280bbc" (UID: "0ac45e3e-179c-4aa8-8a71-5bf626280bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:24.127793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.127718    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume" pod "f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" (UID: "f1b1afb3-5ea7-43f9-9623-5b08b8e0340e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:24.128245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.128218    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:24.128373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.128249    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:24.130652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.130626    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n" pod "0ac45e3e-179c-4aa8-8a71-5bf626280bbc" (UID: "0ac45e3e-179c-4aa8-8a71-5bf626280bbc"). InnerVolumeSpecName "kube-api-access-9rc7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:24.131531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.131505    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td" pod "f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" (UID: "f1b1afb3-5ea7-43f9-9623-5b08b8e0340e"). InnerVolumeSpecName "kube-api-access-qf6td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:24.229901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.229866    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rc7n\" (UniqueName: \"kubernetes.io/projected/0ac45e3e-179c-4aa8-8a71-5bf626280bbc-kube-api-access-9rc7n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:24.229901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.229896    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6td\" (UniqueName: \"kubernetes.io/projected/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e-kube-api-access-qf6td\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:24.248737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.248639    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:24.285710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.284811    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"]
Oct 30 07:19:24.315217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.311518    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ba67d8-fmxbd"]
Oct 30 07:19:24.380248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.380211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:24.399119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.399089    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:19:24.404441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.404415    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba67d8-6f64dfdff5-p277h"]
Oct 30 07:19:24.815579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:24.815491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:25.059149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.059120    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"
Oct 30 07:19:25.115643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.115611    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:19:25.128406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.128376    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba67d8-78c987b4f8-2jpvl"]
Oct 30 07:19:25.691896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.691864    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" path="/var/lib/kubelet/pods/0ac45e3e-179c-4aa8-8a71-5bf626280bbc/volumes"
Oct 30 07:19:25.692394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.692378    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6722aea5-7541-48f7-864e-2e4e7991e20f" path="/var/lib/kubelet/pods/6722aea5-7541-48f7-864e-2e4e7991e20f/volumes"
Oct 30 07:19:25.692781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:25.692767    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" path="/var/lib/kubelet/pods/f1b1afb3-5ea7-43f9-9623-5b08b8e0340e/volumes"
Oct 30 07:19:27.004663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:27.004630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:28.853840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:28.853804    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:29.867855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:29.867820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:29.923663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:29.923634    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:19:29.923846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:29.923702    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:19:30.707837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:30.707801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:31.689526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:31.689486    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:31.800067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:31.800029    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:32.025601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.025515    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:19:32.025819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.025789    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" containerID="cri-o://219c3132dc685c2c633532235d0ee76f949299ae8e99027a68a1a1740fe65709" gracePeriod=30
Oct 30 07:19:32.037377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.037342    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:19:32.037619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.037593    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy" containerID="cri-o://fcc41e4ff22ce58e4ccdb3ff87fb3a85684f5670dce1b8ed19421e37aeac2d7e" gracePeriod=30
Oct 30 07:19:32.061492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.061459    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:32.656690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.656642    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"]
Oct 30 07:19:32.658143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658111    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerName="payload-processing"
Oct 30 07:19:32.658143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658141    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerName="payload-processing"
Oct 30 07:19:32.658362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658176    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerName="payload-pre-processing"
Oct 30 07:19:32.658362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658184    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerName="payload-pre-processing"
Oct 30 07:19:32.658560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658543    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac45e3e-179c-4aa8-8a71-5bf626280bbc" containerName="payload-processing"
Oct 30 07:19:32.658617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.658572    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b1afb3-5ea7-43f9-9623-5b08b8e0340e" containerName="payload-pre-processing"
Oct 30 07:19:32.665222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.665198    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.676279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.676242    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"]
Oct 30 07:19:32.684313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.684268    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:19:32.684526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.684436    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:19:32.685403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.685379    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:19:32.685514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.685420    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:19:32.693464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.693421    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:32.722809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.722777    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:19:32.722985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.722832    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:19:32.790244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.790180    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.791194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.791157    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.791482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.791463    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljt7l\" (UniqueName: \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.893653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.893617    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.893836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.893728    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.893836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.893776    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljt7l\" (UniqueName: \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.894429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.894410    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.896136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.896112    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.917886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.917813    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljt7l\" (UniqueName: \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:32.954357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.954323    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:19:32.983908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:32.983869    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:33.296871 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:33.296844    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72605fd8_1a57_4f11_9516_01895bbc9df5.slice/crio-4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c WatchSource:0}: Error finding container 4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c: Status 404 returned error can't find the container with id 4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c
Oct 30 07:19:33.297391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:33.297369    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"]
Oct 30 07:19:34.053903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:34.053870    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:19:34.169897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:34.169855    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9" event={"ID":"72605fd8-1a57-4f11-9516-01895bbc9df5","Type":"ContainerStarted","Data":"abd2931c278ef8ab6805d3ab2f90419a821a537fc8e863267b475fbe93ba6593"}
Oct 30 07:19:34.170079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:34.169911    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9" event={"ID":"72605fd8-1a57-4f11-9516-01895bbc9df5","Type":"ContainerStarted","Data":"4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c"}
Oct 30 07:19:34.218575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:34.218506    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9" podStartSLOduration=2.218487071 podStartE2EDuration="2.218487071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:34.216413793 +0000 UTC m=+1334.649177690" watchObservedRunningTime="2026-10-30 07:19:34.218487071 +0000 UTC m=+1334.651250966"
Oct 30 07:19:34.902045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:34.902011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:35.028650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.028612    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:19:35.028899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.028850    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" podUID="09954160-f955-4bd7-b032-6f723e09d687" containerName="payload-pre-processing" containerID="cri-o://7443399f1d72e973672bd215ace78ede3e6a93b7104dac5e84505822e1c96c63" gracePeriod=30
Oct 30 07:19:35.109484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.109445    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:35.180053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.179963    6738 generic.go:356] "Generic (PLEG): container finished" podID="72605fd8-1a57-4f11-9516-01895bbc9df5" containerID="abd2931c278ef8ab6805d3ab2f90419a821a537fc8e863267b475fbe93ba6593" exitCode=0
Oct 30 07:19:35.180214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.180151    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9" event={"ID":"72605fd8-1a57-4f11-9516-01895bbc9df5","Type":"ContainerDied","Data":"abd2931c278ef8ab6805d3ab2f90419a821a537fc8e863267b475fbe93ba6593"}
Oct 30 07:19:35.182477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.182451    6738 generic.go:356] "Generic (PLEG): container finished" podID="09954160-f955-4bd7-b032-6f723e09d687" containerID="7443399f1d72e973672bd215ace78ede3e6a93b7104dac5e84505822e1c96c63" exitCode=0
Oct 30 07:19:35.182640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.182497    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" event={"ID":"09954160-f955-4bd7-b032-6f723e09d687","Type":"ContainerDied","Data":"7443399f1d72e973672bd215ace78ede3e6a93b7104dac5e84505822e1c96c63"}
Oct 30 07:19:35.229967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.229933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:35.425320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.420244    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:19:35.542773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.542676    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\" (UniqueName: \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\") pod \"09954160-f955-4bd7-b032-6f723e09d687\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") "
Oct 30 07:19:35.542961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.542834    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\") pod \"09954160-f955-4bd7-b032-6f723e09d687\" (UID: \"09954160-f955-4bd7-b032-6f723e09d687\") "
Oct 30 07:19:35.543332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.543284    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume" pod "09954160-f955-4bd7-b032-6f723e09d687" (UID: "09954160-f955-4bd7-b032-6f723e09d687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:35.543536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.543515    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09954160-f955-4bd7-b032-6f723e09d687-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:35.544931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.544904    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts" pod "09954160-f955-4bd7-b032-6f723e09d687" (UID: "09954160-f955-4bd7-b032-6f723e09d687"). InnerVolumeSpecName "kube-api-access-d8dts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:35.556153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.556129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:35.644956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.644921    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dts\" (UniqueName: \"kubernetes.io/projected/09954160-f955-4bd7-b032-6f723e09d687-kube-api-access-d8dts\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:35.694517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.694475    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:35.737595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.737546    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"]
Oct 30 07:19:35.762317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.755416    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a273fd-pkqt2"]
Oct 30 07:19:35.812126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:35.812015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:36.191215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.191185    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"
Oct 30 07:19:36.191215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.191203    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k" event={"ID":"09954160-f955-4bd7-b032-6f723e09d687","Type":"ContainerDied","Data":"7f9dab65eb5c254d6ef4990be0d9054f8aa3357aa831f6b2598add8bfcc1b703"}
Oct 30 07:19:36.191776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.191303    6738 scope.go:122] "RemoveContainer" containerID="7443399f1d72e973672bd215ace78ede3e6a93b7104dac5e84505822e1c96c63"
Oct 30 07:19:36.247022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.246983    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:19:36.263364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.262686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:36.265497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.265465    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a273fd-77b6449fcf-44z5k"]
Oct 30 07:19:36.345403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.345374    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:36.385025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.384979    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\") pod \"72605fd8-1a57-4f11-9516-01895bbc9df5\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") "
Oct 30 07:19:36.385209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.385194    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\" (UniqueName: \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\") pod \"72605fd8-1a57-4f11-9516-01895bbc9df5\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") "
Oct 30 07:19:36.385273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.385249    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\") pod \"72605fd8-1a57-4f11-9516-01895bbc9df5\" (UID: \"72605fd8-1a57-4f11-9516-01895bbc9df5\") "
Oct 30 07:19:36.385519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.385463    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca" pod "72605fd8-1a57-4f11-9516-01895bbc9df5" (UID: "72605fd8-1a57-4f11-9516-01895bbc9df5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:36.386474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.386448    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:36.388506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.388463    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l" pod "72605fd8-1a57-4f11-9516-01895bbc9df5" (UID: "72605fd8-1a57-4f11-9516-01895bbc9df5"). InnerVolumeSpecName "kube-api-access-ljt7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:36.388756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.388722    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca" pod "72605fd8-1a57-4f11-9516-01895bbc9df5" (UID: "72605fd8-1a57-4f11-9516-01895bbc9df5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:19:36.489232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.489143    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljt7l\" (UniqueName: \"kubernetes.io/projected/72605fd8-1a57-4f11-9516-01895bbc9df5-kube-api-access-ljt7l\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:36.489232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:36.489176    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72605fd8-1a57-4f11-9516-01895bbc9df5-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.204172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.204139    6738 generic.go:356] "Generic (PLEG): container finished" podID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerID="fcc41e4ff22ce58e4ccdb3ff87fb3a85684f5670dce1b8ed19421e37aeac2d7e" exitCode=0
Oct 30 07:19:37.204596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.204214    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerDied","Data":"fcc41e4ff22ce58e4ccdb3ff87fb3a85684f5670dce1b8ed19421e37aeac2d7e"}
Oct 30 07:19:37.204596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.204289    6738 scope.go:122] "RemoveContainer" containerID="f372e1c8d8d4067add08641451a75dc19973043aa94bbe2ae500d67a4226393e"
Oct 30 07:19:37.206432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.206210    6738 generic.go:356] "Generic (PLEG): container finished" podID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerID="219c3132dc685c2c633532235d0ee76f949299ae8e99027a68a1a1740fe65709" exitCode=0
Oct 30 07:19:37.206432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.206360    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" event={"ID":"4677d08e-758f-435f-a3cb-9e61305b1de2","Type":"ContainerDied","Data":"219c3132dc685c2c633532235d0ee76f949299ae8e99027a68a1a1740fe65709"}
Oct 30 07:19:37.208064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.208030    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9" event={"ID":"72605fd8-1a57-4f11-9516-01895bbc9df5","Type":"ContainerDied","Data":"4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c"}
Oct 30 07:19:37.208162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.208066    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb17674266c4dd7aa79dc849388e9539f62893e34989ea30267ab093c31c58c"
Oct 30 07:19:37.208162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.208033    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"
Oct 30 07:19:37.438324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.434174    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:19:37.537372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537327    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.537567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537480    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.537567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537519    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537590    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537670    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537677    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537763    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537814    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537849    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.537882    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\") pod \"4677d08e-758f-435f-a3cb-9e61305b1de2\" (UID: \"4677d08e-758f-435f-a3cb-9e61305b1de2\") "
Oct 30 07:19:37.538780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.538732    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.539111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.539087    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:37.539398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.539370    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.540247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.540221    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.540655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.540613    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.543160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.541534    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.543160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.542848    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "kube-api-access-9nffh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.543684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.543656    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:37.544364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.544273    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy" pod "4677d08e-758f-435f-a3cb-9e61305b1de2" (UID: "4677d08e-758f-435f-a3cb-9e61305b1de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.554265 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:19:37.554214    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:19:37.576041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.576005    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:19:37.639276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639232    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639328    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639383    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639507    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639582    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639615    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639674    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639754    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.639934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.639808    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\") pod \"db4cbf02-a776-4090-8bba-c1d6655028d5\" (UID: \"db4cbf02-a776-4090-8bba-c1d6655028d5\") "
Oct 30 07:19:37.640464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640435    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640466    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640480    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640494    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640508    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640521    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nffh\" (UniqueName: \"kubernetes.io/projected/4677d08e-758f-435f-a3cb-9e61305b1de2-kube-api-access-9nffh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640537    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4677d08e-758f-435f-a3cb-9e61305b1de2-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640550    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4677d08e-758f-435f-a3cb-9e61305b1de2-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.640630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.640472    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:37.643505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.643470    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.643632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.643586    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:37.643841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.643814    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.644024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.644002    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.644207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.644185    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.644410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.644389    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.645954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.645927    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "kube-api-access-22vs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.646595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.646571    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy" pod "db4cbf02-a776-4090-8bba-c1d6655028d5" (UID: "db4cbf02-a776-4090-8bba-c1d6655028d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.692102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.692063    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09954160-f955-4bd7-b032-6f723e09d687" path="/var/lib/kubelet/pods/09954160-f955-4bd7-b032-6f723e09d687/volumes"
Oct 30 07:19:37.692594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.692580    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf51857e-afd3-44ed-a1ed-25b7518d1ced" path="/var/lib/kubelet/pods/cf51857e-afd3-44ed-a1ed-25b7518d1ced/volumes"
Oct 30 07:19:37.741951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.741908    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.741951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.741947    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.741964    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.741980    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.741994    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.742007    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.742019    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22vs9\" (UniqueName: \"kubernetes.io/projected/db4cbf02-a776-4090-8bba-c1d6655028d5-kube-api-access-22vs9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.742034    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4cbf02-a776-4090-8bba-c1d6655028d5-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.742174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:37.742048    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4cbf02-a776-4090-8bba-c1d6655028d5-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:38.039554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.039513    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:38.216964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.216927    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"
Oct 30 07:19:38.216964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.216950    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89" event={"ID":"db4cbf02-a776-4090-8bba-c1d6655028d5","Type":"ContainerDied","Data":"bc7d8f1c5f8b05dcf953eb602c31497f218b9b4f93e7215905dff9f033e5dd4b"}
Oct 30 07:19:38.217533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.217017    6738 scope.go:122] "RemoveContainer" containerID="fcc41e4ff22ce58e4ccdb3ff87fb3a85684f5670dce1b8ed19421e37aeac2d7e"
Oct 30 07:19:38.218822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.218801    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"
Oct 30 07:19:38.218944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.218836    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd" event={"ID":"4677d08e-758f-435f-a3cb-9e61305b1de2","Type":"ContainerDied","Data":"f8e8d72c1b332eeaa1609b949980ab782e3ff41f07a09ec0edefe74f9537cd7c"}
Oct 30 07:19:38.228572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.228545    6738 scope.go:122] "RemoveContainer" containerID="219c3132dc685c2c633532235d0ee76f949299ae8e99027a68a1a1740fe65709"
Oct 30 07:19:38.270509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.270475    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:19:38.282767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.282733    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-57h89"]
Oct 30 07:19:38.313613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.313505    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:19:38.322835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:38.322797    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-l4jxd"]
Oct 30 07:19:39.691231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:39.691197    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" path="/var/lib/kubelet/pods/4677d08e-758f-435f-a3cb-9e61305b1de2/volumes"
Oct 30 07:19:39.691878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:39.691858    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" path="/var/lib/kubelet/pods/db4cbf02-a776-4090-8bba-c1d6655028d5/volumes"
Oct 30 07:19:39.893120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:39.893086    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:40.010483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.009285    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:40.126131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.126098    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:40.621677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.621649    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"]
Oct 30 07:19:40.622463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622433    6738 cpu_manager.go:414] "containerMap: removing container" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.622463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622460    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622480    6738 cpu_manager.go:414] "containerMap: removing container" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622488    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622532    6738 cpu_manager.go:414] "containerMap: removing container" podUID="72605fd8-1a57-4f11-9516-01895bbc9df5" containerName="revoke-keys"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622541    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72605fd8-1a57-4f11-9516-01895bbc9df5" containerName="revoke-keys"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622556    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622564    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622583    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09954160-f955-4bd7-b032-6f723e09d687" containerName="payload-pre-processing"
Oct 30 07:19:40.622664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622592    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09954160-f955-4bd7-b032-6f723e09d687" containerName="payload-pre-processing"
Oct 30 07:19:40.623019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622733    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="72605fd8-1a57-4f11-9516-01895bbc9df5" containerName="revoke-keys"
Oct 30 07:19:40.623019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622750    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4677d08e-758f-435f-a3cb-9e61305b1de2" containerName="istio-proxy"
Oct 30 07:19:40.623019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622760    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09954160-f955-4bd7-b032-6f723e09d687" containerName="payload-pre-processing"
Oct 30 07:19:40.623019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622773    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.623019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.622781    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4cbf02-a776-4090-8bba-c1d6655028d5" containerName="istio-proxy"
Oct 30 07:19:40.626351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.626331    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.644367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.644333    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"]
Oct 30 07:19:40.653338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.652725    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:19:40.653338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.652852    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:19:40.653338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.652885    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:19:40.656086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.656049    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:19:40.779701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.779665    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.780193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.779708    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.780193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.779815    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9b6\" (UniqueName: \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.881803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.881712    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.881803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.881759    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.881803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.881795    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9b6\" (UniqueName: \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.882483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.882455    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.884046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.884017    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.906372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.906342    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9b6\" (UniqueName: \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\") pod \"maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:40.939324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:40.939276    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:41.215393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:41.215285    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:19:41.300987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:41.300952    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"]
Oct 30 07:19:41.301391 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:19:41.301365    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc79d6d_9263_42f7_b2d7_66420000c9de.slice/crio-6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1 WatchSource:0}: Error finding container 6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1: Status 404 returned error can't find the container with id 6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1
Oct 30 07:19:41.302206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:41.302098    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:42.027527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:42.027491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:42.255027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:42.254982    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86" event={"ID":"8fc79d6d-9263-42f7-b2d7-66420000c9de","Type":"ContainerStarted","Data":"6a0e7c1718f0d76191750548b7439b65ad51f21c14e79cae3221b0a8bd6a01d5"}
Oct 30 07:19:42.255348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:42.255049    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86" event={"ID":"8fc79d6d-9263-42f7-b2d7-66420000c9de","Type":"ContainerStarted","Data":"6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1"}
Oct 30 07:19:42.297691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:42.297568    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86" podStartSLOduration=2.297553674 podStartE2EDuration="2.297553674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:42.295440718 +0000 UTC m=+1342.728204611" watchObservedRunningTime="2026-10-30 07:19:42.297553674 +0000 UTC m=+1342.730317571"
Oct 30 07:19:43.267526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:43.267490    6738 generic.go:356] "Generic (PLEG): container finished" podID="8fc79d6d-9263-42f7-b2d7-66420000c9de" containerID="6a0e7c1718f0d76191750548b7439b65ad51f21c14e79cae3221b0a8bd6a01d5" exitCode=0
Oct 30 07:19:43.267977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:43.267569    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86" event={"ID":"8fc79d6d-9263-42f7-b2d7-66420000c9de","Type":"ContainerDied","Data":"6a0e7c1718f0d76191750548b7439b65ad51f21c14e79cae3221b0a8bd6a01d5"}
Oct 30 07:19:43.899958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:43.899917    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:44.437206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.437176    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:44.570769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.570671    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\") pod \"8fc79d6d-9263-42f7-b2d7-66420000c9de\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") "
Oct 30 07:19:44.570965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.570911    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\" (UniqueName: \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\") pod \"8fc79d6d-9263-42f7-b2d7-66420000c9de\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") "
Oct 30 07:19:44.570965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.570946    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\") pod \"8fc79d6d-9263-42f7-b2d7-66420000c9de\" (UID: \"8fc79d6d-9263-42f7-b2d7-66420000c9de\") "
Oct 30 07:19:44.571384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.571357    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca" pod "8fc79d6d-9263-42f7-b2d7-66420000c9de" (UID: "8fc79d6d-9263-42f7-b2d7-66420000c9de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:44.572838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.572809    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca" pod "8fc79d6d-9263-42f7-b2d7-66420000c9de" (UID: "8fc79d6d-9263-42f7-b2d7-66420000c9de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:19:44.573203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.573183    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6" pod "8fc79d6d-9263-42f7-b2d7-66420000c9de" (UID: "8fc79d6d-9263-42f7-b2d7-66420000c9de"). InnerVolumeSpecName "kube-api-access-7q9b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:44.672485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.672451    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9b6\" (UniqueName: \"kubernetes.io/projected/8fc79d6d-9263-42f7-b2d7-66420000c9de-kube-api-access-7q9b6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:44.672485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.672481    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:44.672485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:44.672494    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc79d6d-9263-42f7-b2d7-66420000c9de-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:45.285886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:45.285853    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"
Oct 30 07:19:45.286072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:45.285888    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86" event={"ID":"8fc79d6d-9263-42f7-b2d7-66420000c9de","Type":"ContainerDied","Data":"6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1"}
Oct 30 07:19:45.286072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:45.285934    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad5858d170adeb36fa9a6172cc59e23bae37d1ca26304265a40159553cd63a1"
Oct 30 07:19:45.790131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:45.790074    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:47.962631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:47.962584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:48.065628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.065589    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:19:48.065825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.065803    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" podUID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" containerName="payload-processing" containerID="cri-o://a4ef97e9fc64c58949718bd4a758d3c6bdbe601aa4d89b45f93cda4044e443b2" gracePeriod=30
Oct 30 07:19:48.087465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.087430    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:19:48.087719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.087685    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" podUID="70599fb1-d59f-4905-8bae-8cee724f92af" containerName="payload-pre-processing" containerID="cri-o://34dc5e69ba4bae40c8214d907132715591cf6967eb86b65da0babd1771abc777" gracePeriod=30
Oct 30 07:19:48.157683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.157589    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:48.272717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.272537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:48.317928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.317891    6738 generic.go:356] "Generic (PLEG): container finished" podID="70599fb1-d59f-4905-8bae-8cee724f92af" containerID="34dc5e69ba4bae40c8214d907132715591cf6967eb86b65da0babd1771abc777" exitCode=0
Oct 30 07:19:48.318149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.317961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" event={"ID":"70599fb1-d59f-4905-8bae-8cee724f92af","Type":"ContainerDied","Data":"34dc5e69ba4bae40c8214d907132715591cf6967eb86b65da0babd1771abc777"}
Oct 30 07:19:48.320388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.320359    6738 generic.go:356] "Generic (PLEG): container finished" podID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" containerID="a4ef97e9fc64c58949718bd4a758d3c6bdbe601aa4d89b45f93cda4044e443b2" exitCode=0
Oct 30 07:19:48.320554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.320436    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" event={"ID":"97ead4a7-92a2-4441-88a3-fc5bc27f0a39","Type":"ContainerDied","Data":"a4ef97e9fc64c58949718bd4a758d3c6bdbe601aa4d89b45f93cda4044e443b2"}
Oct 30 07:19:48.552322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.550996    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:19:48.558607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.558584    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:19:48.584511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.584402    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:48.727685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.727649    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\") pod \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") "
Oct 30 07:19:48.727902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.727697    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\" (UniqueName: \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\") pod \"70599fb1-d59f-4905-8bae-8cee724f92af\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") "
Oct 30 07:19:48.727982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.727900    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\" (UniqueName: \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\") pod \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\" (UID: \"97ead4a7-92a2-4441-88a3-fc5bc27f0a39\") "
Oct 30 07:19:48.728046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.728001    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\") pod \"70599fb1-d59f-4905-8bae-8cee724f92af\" (UID: \"70599fb1-d59f-4905-8bae-8cee724f92af\") "
Oct 30 07:19:48.728162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.728140    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume" pod "97ead4a7-92a2-4441-88a3-fc5bc27f0a39" (UID: "97ead4a7-92a2-4441-88a3-fc5bc27f0a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:48.728443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.728410    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume" pod "70599fb1-d59f-4905-8bae-8cee724f92af" (UID: "70599fb1-d59f-4905-8bae-8cee724f92af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:48.728655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.728642    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70599fb1-d59f-4905-8bae-8cee724f92af-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:48.728705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.728659    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:48.730040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.730019    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4" pod "70599fb1-d59f-4905-8bae-8cee724f92af" (UID: "70599fb1-d59f-4905-8bae-8cee724f92af"). InnerVolumeSpecName "kube-api-access-fcvr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:48.730887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.730870    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m" pod "97ead4a7-92a2-4441-88a3-fc5bc27f0a39" (UID: "97ead4a7-92a2-4441-88a3-fc5bc27f0a39"). InnerVolumeSpecName "kube-api-access-h4d8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:48.848533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.848244    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:48.855026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.854414    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvr4\" (UniqueName: \"kubernetes.io/projected/70599fb1-d59f-4905-8bae-8cee724f92af-kube-api-access-fcvr4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:48.855026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.854450    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4d8m\" (UniqueName: \"kubernetes.io/projected/97ead4a7-92a2-4441-88a3-fc5bc27f0a39-kube-api-access-h4d8m\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:48.919184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.919137    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"]
Oct 30 07:19:48.924267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:48.924234    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1f203-8gvw9"]
Oct 30 07:19:49.051672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.051630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:49.329789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.329758    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"
Oct 30 07:19:49.329972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.329794    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64" event={"ID":"70599fb1-d59f-4905-8bae-8cee724f92af","Type":"ContainerDied","Data":"b4170c87d9859176b3cad78f886202172cc49c1d85328859139899a94408eda8"}
Oct 30 07:19:49.329972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.329854    6738 scope.go:122] "RemoveContainer" containerID="34dc5e69ba4bae40c8214d907132715591cf6967eb86b65da0babd1771abc777"
Oct 30 07:19:49.331485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.331453    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9" event={"ID":"97ead4a7-92a2-4441-88a3-fc5bc27f0a39","Type":"ContainerDied","Data":"781edf65da7a71610ef2c85964e63f91de171f0248bb9a81ff52f617f13795b3"}
Oct 30 07:19:49.331563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.331490    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"
Oct 30 07:19:49.341826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.341800    6738 scope.go:122] "RemoveContainer" containerID="a4ef97e9fc64c58949718bd4a758d3c6bdbe601aa4d89b45f93cda4044e443b2"
Oct 30 07:19:49.370455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.370420    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:49.384233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.384201    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:19:49.389180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.389147    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-7b6c968f47-b8g64"]
Oct 30 07:19:49.431064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.431028    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:19:49.434174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.434144    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-59f7d8f458-d2ck9"]
Oct 30 07:19:49.690407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.690377    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70599fb1-d59f-4905-8bae-8cee724f92af" path="/var/lib/kubelet/pods/70599fb1-d59f-4905-8bae-8cee724f92af/volumes"
Oct 30 07:19:49.690849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.690836    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72605fd8-1a57-4f11-9516-01895bbc9df5" path="/var/lib/kubelet/pods/72605fd8-1a57-4f11-9516-01895bbc9df5/volumes"
Oct 30 07:19:49.691200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:49.691189    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" path="/var/lib/kubelet/pods/97ead4a7-92a2-4441-88a3-fc5bc27f0a39/volumes"
Oct 30 07:19:51.539700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:51.539667    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:53.457152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:53.457107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:55.329076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:55.329043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:56.235740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.235702    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:56.243349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.243316    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" podUID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" containerName="main" containerID="cri-o://569216576cd988df6a610886bc079c7b39e83c80661e2609ff73a8b1b584c0b7" gracePeriod=45
Oct 30 07:19:56.357402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.357288    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:56.443889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.443855    6738 generic.go:356] "Generic (PLEG): container finished" podID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" containerID="569216576cd988df6a610886bc079c7b39e83c80661e2609ff73a8b1b584c0b7" exitCode=0
Oct 30 07:19:56.444083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.443922    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" event={"ID":"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1","Type":"ContainerDied","Data":"569216576cd988df6a610886bc079c7b39e83c80661e2609ff73a8b1b584c0b7"}
Oct 30 07:19:56.505829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.505800    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:19:56.578740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.578562    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.579546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579069    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\" (UniqueName: \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.579546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579159    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.579546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579231    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.579546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579279    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.579546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579403    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\") pod \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\" (UID: \"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1\") "
Oct 30 07:19:56.580005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579958    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:56.580005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.579969    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:56.580192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.580011    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:56.580530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.580490    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:56.580530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.580518    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:56.580687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.580532    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:56.583142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.583117    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:19:56.584107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.584085    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:19:56.584353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.584325    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:56.584477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.584350    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" containerID="cri-o://d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c" gracePeriod=30
Oct 30 07:19:56.584908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.584886    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n" pod "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" (UID: "8e49d9ec-ceae-4a6d-8a63-9690b5c429f1"). InnerVolumeSpecName "kube-api-access-gfd8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:56.607716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.607674    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:56.754319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.754132    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfd8n\" (UniqueName: \"kubernetes.io/projected/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-kube-api-access-gfd8n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:56.754319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.754169    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:56.754319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:56.754187    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:19:57.453564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.453533    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"
Oct 30 07:19:57.454154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.453565    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk" event={"ID":"8e49d9ec-ceae-4a6d-8a63-9690b5c429f1","Type":"ContainerDied","Data":"a6ed400ee626c306663b55238baece850faa2b1780847d641c8d26c66f7e380c"}
Oct 30 07:19:57.454154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.453627    6738 scope.go:122] "RemoveContainer" containerID="569216576cd988df6a610886bc079c7b39e83c80661e2609ff73a8b1b584c0b7"
Oct 30 07:19:57.458346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.458272    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:57.534503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.534471    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:19:57.538611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.538572    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a273fd/ipp-route-a273fd-kserve-578b47dbb9-wxxxk"]
Oct 30 07:19:57.691676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:57.691628    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" path="/var/lib/kubelet/pods/8e49d9ec-ceae-4a6d-8a63-9690b5c429f1/volumes"
Oct 30 07:19:59.483729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.483630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:59.838676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.838583    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:19:59.929403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.929367    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:19:59.930284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930262    6738 cpu_manager.go:414] "containerMap: removing container" podUID="70599fb1-d59f-4905-8bae-8cee724f92af" containerName="payload-pre-processing"
Oct 30 07:19:59.930284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930283    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70599fb1-d59f-4905-8bae-8cee724f92af" containerName="payload-pre-processing"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930328    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8fc79d6d-9263-42f7-b2d7-66420000c9de" containerName="revoke-keys"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930336    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc79d6d-9263-42f7-b2d7-66420000c9de" containerName="revoke-keys"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930365    6738 cpu_manager.go:414] "containerMap: removing container" podUID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" containerName="payload-processing"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930370    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" containerName="payload-processing"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930380    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" containerName="main"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930385    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" containerName="main"
Oct 30 07:19:59.930506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930507    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="70599fb1-d59f-4905-8bae-8cee724f92af" containerName="payload-pre-processing"
Oct 30 07:19:59.930970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930521    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc79d6d-9263-42f7-b2d7-66420000c9de" containerName="revoke-keys"
Oct 30 07:19:59.930970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930534    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ead4a7-92a2-4441-88a3-fc5bc27f0a39" containerName="payload-processing"
Oct 30 07:19:59.930970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.930541    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e49d9ec-ceae-4a6d-8a63-9690b5c429f1" containerName="main"
Oct 30 07:19:59.933964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.933943    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:19:59.964618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.964584    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b263d4-openshift-default-dockercfg-x9fxb\""
Oct 30 07:19:59.968018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.967984    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:19:59.994439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:19:59.994403    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104164    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104283    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104368    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104450    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104531    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104629    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104653    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvhd\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.108510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.104699    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.149937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.138882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:00.212499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212158    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212264    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212373    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212413    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvhd\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212450    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212534    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212568    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.212960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.212657    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.213494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.213466    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.213835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.213732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.216513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.216488    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.216737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.216533    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.216935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.216912    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.220164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.219652    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.222686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.222658    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.249085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.249044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvhd\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.267213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.267176    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\") pod \"e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") " pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.548984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.548944    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:00.672261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.672222    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:20:00.672506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.672285    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:20:00.881910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.881546    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:20:00.882119 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:00.882094    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d550f8_5ef4_473e_a0e4_4d005b40ee8c.slice/crio-a3ea42fd0fd8c8513cbf0ad5573e136adf7365bcc8bcc144ba7e0c1ab3f595ed WatchSource:0}: Error finding container a3ea42fd0fd8c8513cbf0ad5573e136adf7365bcc8bcc144ba7e0c1ab3f595ed: Status 404 returned error can't find the container with id a3ea42fd0fd8c8513cbf0ad5573e136adf7365bcc8bcc144ba7e0c1ab3f595ed
Oct 30 07:20:00.885096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.885054    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:00.885189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.885142    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:00.885227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:00.885189    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:01.488212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.488176    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerStarted","Data":"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"}
Oct 30 07:20:01.488458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.488265    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerStarted","Data":"a3ea42fd0fd8c8513cbf0ad5573e136adf7365bcc8bcc144ba7e0c1ab3f595ed"}
Oct 30 07:20:01.549058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.548988    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podStartSLOduration=2.548973204 podStartE2EDuration="2.548973204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:01.545636429 +0000 UTC m=+1361.978400322" watchObservedRunningTime="2026-10-30 07:20:01.548973204 +0000 UTC m=+1361.981737098"
Oct 30 07:20:01.549522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.549195    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:01.555251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.555220    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:01.629793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.629755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:01.917267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.917240    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:20:01.942804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.942958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942819    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.942958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942841    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.942958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942901    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.942958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942937    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.943206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.942976    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.943206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943022    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.943206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943076    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:01.943206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943129    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:01.943206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943179    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.943516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943219    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\") pod \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\" (UID: \"bf7cefa8-8775-44f6-8b8d-3170c6b80303\") "
Oct 30 07:20:01.943516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943331    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:01.943516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943385    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:01.943669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943583    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:01.943907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943886    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:01.944004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943912    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:01.944004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943925    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:01.944004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943938    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:01.944004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.943952    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:01.945510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.945480    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:01.945616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.945550    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "kube-api-access-zvhrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:01.945695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.945676    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:01.945732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:01.945673    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy" pod "bf7cefa8-8775-44f6-8b8d-3170c6b80303" (UID: "bf7cefa8-8775-44f6-8b8d-3170c6b80303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:02.045887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.045799    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:02.045887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.045832    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:02.045887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.045840    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7cefa8-8775-44f6-8b8d-3170c6b80303-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:02.045887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.045849    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhrp\" (UniqueName: \"kubernetes.io/projected/bf7cefa8-8775-44f6-8b8d-3170c6b80303-kube-api-access-zvhrp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:02.496990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.496952    6738 generic.go:356] "Generic (PLEG): container finished" podID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerID="d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c" exitCode=0
Oct 30 07:20:02.497209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.497037    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"
Oct 30 07:20:02.497209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.497041    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerDied","Data":"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"}
Oct 30 07:20:02.497209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.497120    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr" event={"ID":"bf7cefa8-8775-44f6-8b8d-3170c6b80303","Type":"ContainerDied","Data":"0b91723346ce4cd8fa217924a3bec17caec49cd2966638542495c7f6fb77a5db"}
Oct 30 07:20:02.497209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.497158    6738 scope.go:122] "RemoveContainer" containerID="d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"
Oct 30 07:20:02.497728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.497627    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:02.498787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.498766    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:02.509118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.509092    6738 scope.go:122] "RemoveContainer" containerID="534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"
Oct 30 07:20:02.520201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.520180    6738 scope.go:122] "RemoveContainer" containerID="d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"
Oct 30 07:20:02.520558 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:02.520539    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c\": container with ID starting with d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c not found: ID does not exist" containerID="d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"
Oct 30 07:20:02.520614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.520566    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c"} err="failed to get container status \"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c\": rpc error: code = NotFound desc = could not find container \"d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c\": container with ID starting with d864aadc06d7a7b8d2c2a39695012e084b474ba3995087a13d248e87b899894c not found: ID does not exist"
Oct 30 07:20:02.520614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.520584    6738 scope.go:122] "RemoveContainer" containerID="534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"
Oct 30 07:20:02.520862 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:02.520845    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f\": container with ID starting with 534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f not found: ID does not exist" containerID="534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"
Oct 30 07:20:02.520911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.520868    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f"} err="failed to get container status \"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f\": rpc error: code = NotFound desc = could not find container \"534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f\": container with ID starting with 534c73f660f5701ae6499e0529d4aa2bef119297f427b6bf314fdf1d05e0880f not found: ID does not exist"
Oct 30 07:20:02.615341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.615274    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:20:02.619502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:02.619480    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1f203-openshift-default-f89bc77ff-5x9xr"]
Oct 30 07:20:03.436103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:03.436067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:03.690775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:03.690694    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" path="/var/lib/kubelet/pods/bf7cefa8-8775-44f6-8b8d-3170c6b80303/volumes"
Oct 30 07:20:03.948003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:03.947907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:04.063576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:04.063541    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:05.622850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:05.622811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:06.400677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.400621    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:06.510600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.510565    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:06.536469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.536430    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:20:06.537135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537117    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.537135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537135    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.537259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537158    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.537259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537163    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.537347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537277    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.537347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.537313    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7cefa8-8775-44f6-8b8d-3170c6b80303" containerName="istio-proxy"
Oct 30 07:20:06.540918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.540898    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.558692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.558655    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:20:06.567192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.567155    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b263d4\""
Oct 30 07:20:06.614485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.614442    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2x8\" (UniqueName: \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.614689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.614609    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.616218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.616192    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:20:06.619884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.619861    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.637414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.637382    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b263d4-dockercfg-k4xfn\""
Oct 30 07:20:06.642233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.642194    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:20:06.725424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.725321    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.725424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.725392    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqw2\" (UniqueName: \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.725663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.725506    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2x8\" (UniqueName: \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.725663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.725553    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.726548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.726517    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:06.859327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.856672    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.910949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.910265    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.910949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.910464    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqw2\" (UniqueName: \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.943512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.943475    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqw2\" (UniqueName: \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\") pod \"payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:06.944278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:06.944255    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2x8\" (UniqueName: \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\") pod \"payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:07.156062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.156029    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:07.238513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.237531    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:07.496823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.496795    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:20:07.498584 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:07.498552    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0f9430_dce1_454c_9a4c_f4faadbb90ff.slice/crio-c6d14d24ca12c3dc5d86e61368eba1d4b6eb915370d5515dcb5f2c49d4fcc02b WatchSource:0}: Error finding container c6d14d24ca12c3dc5d86e61368eba1d4b6eb915370d5515dcb5f2c49d4fcc02b: Status 404 returned error can't find the container with id c6d14d24ca12c3dc5d86e61368eba1d4b6eb915370d5515dcb5f2c49d4fcc02b
Oct 30 07:20:07.539622 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:07.539588    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:20:07.559094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.559058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" event={"ID":"2f0f9430-dce1-454c-9a4c-f4faadbb90ff","Type":"ContainerStarted","Data":"c6d14d24ca12c3dc5d86e61368eba1d4b6eb915370d5515dcb5f2c49d4fcc02b"}
Oct 30 07:20:07.647328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.647280    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:20:07.648462 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:07.648437    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f102f5_3591_4fc2_b341_bf7f3e1df095.slice/crio-4a458d465cbd438839f18762050dc38afadc0f5e478c57d21620eae399866c59 WatchSource:0}: Error finding container 4a458d465cbd438839f18762050dc38afadc0f5e478c57d21620eae399866c59: Status 404 returned error can't find the container with id 4a458d465cbd438839f18762050dc38afadc0f5e478c57d21620eae399866c59
Oct 30 07:20:07.906441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:07.906400    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:08.594464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.594410    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" event={"ID":"2f0f9430-dce1-454c-9a4c-f4faadbb90ff","Type":"ContainerStarted","Data":"93efc8bb9781fefd7e809c1215447d8a1459bf9ee7de40fb8dc19cc6c5be031d"}
Oct 30 07:20:08.594904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.594880    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:08.596653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.596626    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" event={"ID":"f2f102f5-3591-4fc2-b341-bf7f3e1df095","Type":"ContainerStarted","Data":"16e0e8e223f4e633e8dcf978ca546211238b787d13d9698131a5cf86ac5df856"}
Oct 30 07:20:08.596788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.596670    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" event={"ID":"f2f102f5-3591-4fc2-b341-bf7f3e1df095","Type":"ContainerStarted","Data":"4a458d465cbd438839f18762050dc38afadc0f5e478c57d21620eae399866c59"}
Oct 30 07:20:08.596788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.596697    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:08.639600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.639542    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" podStartSLOduration=2.639528055 podStartE2EDuration="2.639528055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:08.63765582 +0000 UTC m=+1369.070419713" watchObservedRunningTime="2026-10-30 07:20:08.639528055 +0000 UTC m=+1369.072291996"
Oct 30 07:20:08.683958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:08.683286    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" podStartSLOduration=2.6832642780000002 podStartE2EDuration="2.683264278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:08.679909217 +0000 UTC m=+1369.112673113" watchObservedRunningTime="2026-10-30 07:20:08.683264278 +0000 UTC m=+1369.116028176"
Oct 30 07:20:09.964201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:09.964167    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:12.183206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.183162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:12.279241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.279202    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:20:12.297198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.297158    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:20:12.297424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.297395    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" podUID="48b35560-06c5-401c-a0b3-ec16999dcbd0" containerName="payload-pre-processing" containerID="cri-o://5fd9ae74f588f6a922f6a650450244f0d2b2369bb400334eb94ecda1c4f105d3" gracePeriod=30
Oct 30 07:20:12.370384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.370273    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:12.491768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.491609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:12.719385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.717450    6738 generic.go:356] "Generic (PLEG): container finished" podID="48b35560-06c5-401c-a0b3-ec16999dcbd0" containerID="5fd9ae74f588f6a922f6a650450244f0d2b2369bb400334eb94ecda1c4f105d3" exitCode=0
Oct 30 07:20:12.719385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.717790    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" podUID="307eb1bf-09ed-4d5d-adc6-005e994af3de" containerName="payload-processing" containerID="cri-o://ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74" gracePeriod=30
Oct 30 07:20:12.719385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.718133    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" event={"ID":"48b35560-06c5-401c-a0b3-ec16999dcbd0","Type":"ContainerDied","Data":"5fd9ae74f588f6a922f6a650450244f0d2b2369bb400334eb94ecda1c4f105d3"}
Oct 30 07:20:12.780105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.778255    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:20:12.822896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.822857    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\") pod \"48b35560-06c5-401c-a0b3-ec16999dcbd0\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") "
Oct 30 07:20:12.823123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.823102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\" (UniqueName: \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\") pod \"48b35560-06c5-401c-a0b3-ec16999dcbd0\" (UID: \"48b35560-06c5-401c-a0b3-ec16999dcbd0\") "
Oct 30 07:20:12.824179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.824147    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume" pod "48b35560-06c5-401c-a0b3-ec16999dcbd0" (UID: "48b35560-06c5-401c-a0b3-ec16999dcbd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:12.833104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.833070    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c" pod "48b35560-06c5-401c-a0b3-ec16999dcbd0" (UID: "48b35560-06c5-401c-a0b3-ec16999dcbd0"). InnerVolumeSpecName "kube-api-access-n4h2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:12.861020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.860985    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:20:12.861866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.861839    6738 cpu_manager.go:414] "containerMap: removing container" podUID="48b35560-06c5-401c-a0b3-ec16999dcbd0" containerName="payload-pre-processing"
Oct 30 07:20:12.861866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.861863    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b35560-06c5-401c-a0b3-ec16999dcbd0" containerName="payload-pre-processing"
Oct 30 07:20:12.862069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.862034    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b35560-06c5-401c-a0b3-ec16999dcbd0" containerName="payload-pre-processing"
Oct 30 07:20:12.864883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.864862    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.883544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.883160    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a51baa\"/\"praxis-route-a51baa-kserve-self-signed-certs\""
Oct 30 07:20:12.884589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.883796    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a51baa\"/\"openshift-service-ca.crt\""
Oct 30 07:20:12.884589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.884471    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a51baa\"/\"kube-root-ca.crt\""
Oct 30 07:20:12.884589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.884544    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a51baa\"/\"default-dockercfg-b8rwf\""
Oct 30 07:20:12.889613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.889589    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:20:12.896819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.896786    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926114    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926181    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926268    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926319    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926365    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjt7k\" (UniqueName: \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926467    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926690    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b35560-06c5-401c-a0b3-ec16999dcbd0-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:12.934359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:12.926710    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4h2c\" (UniqueName: \"kubernetes.io/projected/48b35560-06c5-401c-a0b3-ec16999dcbd0-kube-api-access-n4h2c\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:13.014914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.014872    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:13.030264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030163    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030222    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030348    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030380    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030425    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjt7k\" (UniqueName: \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030502    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.030946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.030924    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.031595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.031567    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.031749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.031635    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.034033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.034010    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.034224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.034197    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.058139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.058069    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"]
Oct 30 07:20:13.060502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.060465    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjt7k\" (UniqueName: \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\") pod \"praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") " pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.061194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.061168    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-137ec6-mdb86"]
Oct 30 07:20:13.137078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.137033    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:20:13.146314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.141153    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:13.203965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.199219    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:13.252673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.252162    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\") pod \"307eb1bf-09ed-4d5d-adc6-005e994af3de\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") "
Oct 30 07:20:13.253326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.252951    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume" pod "307eb1bf-09ed-4d5d-adc6-005e994af3de" (UID: "307eb1bf-09ed-4d5d-adc6-005e994af3de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:13.253326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.252645    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\" (UniqueName: \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\") pod \"307eb1bf-09ed-4d5d-adc6-005e994af3de\" (UID: \"307eb1bf-09ed-4d5d-adc6-005e994af3de\") "
Oct 30 07:20:13.254174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.254140    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307eb1bf-09ed-4d5d-adc6-005e994af3de-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:13.255280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.255257    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb" pod "307eb1bf-09ed-4d5d-adc6-005e994af3de" (UID: "307eb1bf-09ed-4d5d-adc6-005e994af3de"). InnerVolumeSpecName "kube-api-access-5rrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:13.355335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.355247    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rrtb\" (UniqueName: \"kubernetes.io/projected/307eb1bf-09ed-4d5d-adc6-005e994af3de-kube-api-access-5rrtb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:13.583081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.583049    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:20:13.583836 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:13.583808    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b68e9da_e0cf_4a6c_a468_86a861ec92e1.slice/crio-9922f61e4ee233f50b96f89d0ca6e38c48f6b22745e0641b74f1ab0546f7214d WatchSource:0}: Error finding container 9922f61e4ee233f50b96f89d0ca6e38c48f6b22745e0641b74f1ab0546f7214d: Status 404 returned error can't find the container with id 9922f61e4ee233f50b96f89d0ca6e38c48f6b22745e0641b74f1ab0546f7214d
Oct 30 07:20:13.618629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.618589    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:13.692887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.692857    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc79d6d-9263-42f7-b2d7-66420000c9de" path="/var/lib/kubelet/pods/8fc79d6d-9263-42f7-b2d7-66420000c9de/volumes"
Oct 30 07:20:13.728718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.728368    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564" event={"ID":"48b35560-06c5-401c-a0b3-ec16999dcbd0","Type":"ContainerDied","Data":"b868bc6b84e6c64ad57b3cf8a1a1f7aded7bcc1ad53e521ff3abb1d184a9e2cf"}
Oct 30 07:20:13.728718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.728460    6738 scope.go:122] "RemoveContainer" containerID="5fd9ae74f588f6a922f6a650450244f0d2b2369bb400334eb94ecda1c4f105d3"
Oct 30 07:20:13.728718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.728482    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"
Oct 30 07:20:13.730676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.730649    6738 generic.go:356] "Generic (PLEG): container finished" podID="307eb1bf-09ed-4d5d-adc6-005e994af3de" containerID="ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74" exitCode=0
Oct 30 07:20:13.730895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.730873    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" event={"ID":"307eb1bf-09ed-4d5d-adc6-005e994af3de","Type":"ContainerDied","Data":"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"}
Oct 30 07:20:13.730974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.730927    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5" event={"ID":"307eb1bf-09ed-4d5d-adc6-005e994af3de","Type":"ContainerDied","Data":"f0aed21e7ee147b2263fa55f9813c5744293b8cb38c3ff9356b0f35b84252edb"}
Oct 30 07:20:13.731034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.730997    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"
Oct 30 07:20:13.734152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.733842    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" event={"ID":"0b68e9da-e0cf-4a6c-a468-86a861ec92e1","Type":"ContainerStarted","Data":"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"}
Oct 30 07:20:13.734152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.733882    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" event={"ID":"0b68e9da-e0cf-4a6c-a468-86a861ec92e1","Type":"ContainerStarted","Data":"9922f61e4ee233f50b96f89d0ca6e38c48f6b22745e0641b74f1ab0546f7214d"}
Oct 30 07:20:13.742009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.741981    6738 scope.go:122] "RemoveContainer" containerID="ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"
Oct 30 07:20:13.754029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.754007    6738 scope.go:122] "RemoveContainer" containerID="ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"
Oct 30 07:20:13.754427 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:13.754406    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74\": container with ID starting with ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74 not found: ID does not exist" containerID="ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"
Oct 30 07:20:13.754510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.754440    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74"} err="failed to get container status \"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74\": rpc error: code = NotFound desc = could not find container \"ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74\": container with ID starting with ffb574ecc662c0469b87ca72380a6d8c2842acaf52459001651334b24c11ec74 not found: ID does not exist"
Oct 30 07:20:13.781532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.781493    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:20:13.786005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.785972    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-137ec6-6cc676c9d7-rl564"]
Oct 30 07:20:13.819940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.819910    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:20:13.823150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.823122    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-137ec6-57465464f6-vvgx5"]
Oct 30 07:20:13.866008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:13.865905    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" podStartSLOduration=1.865886583 podStartE2EDuration="1.865886583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:13.861869714 +0000 UTC m=+1374.294633607" watchObservedRunningTime="2026-10-30 07:20:13.865886583 +0000 UTC m=+1374.298650480"
Oct 30 07:20:14.204744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:14.204706    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:14.876579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:14.876540    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:20:14.876818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:14.876791    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy" containerID="cri-o://b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763" gracePeriod=30
Oct 30 07:20:15.060688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.060273    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": read tcp 10.133.0.2:58738->10.133.0.176:15021: read: connection reset by peer" start-of-body=
Oct 30 07:20:15.060688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.060369    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": read tcp 10.133.0.2:58738->10.133.0.176:15021: read: connection reset by peer"
Oct 30 07:20:15.230623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.230512    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:15.690649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.690614    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307eb1bf-09ed-4d5d-adc6-005e994af3de" path="/var/lib/kubelet/pods/307eb1bf-09ed-4d5d-adc6-005e994af3de/volumes"
Oct 30 07:20:15.691044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.691032    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b35560-06c5-401c-a0b3-ec16999dcbd0" path="/var/lib/kubelet/pods/48b35560-06c5-401c-a0b3-ec16999dcbd0/volumes"
Oct 30 07:20:15.757450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.757411    6738 generic.go:356] "Generic (PLEG): container finished" podID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerID="fab15fbcee5b3b59fef0ff4b9949243cb21c6ebdf5b3c075e8197f774d2ca551" exitCode=0
Oct 30 07:20:15.757662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.757484    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerDied","Data":"fab15fbcee5b3b59fef0ff4b9949243cb21c6ebdf5b3c075e8197f774d2ca551"}
Oct 30 07:20:15.758004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.757986    6738 scope.go:122] "RemoveContainer" containerID="fab15fbcee5b3b59fef0ff4b9949243cb21c6ebdf5b3c075e8197f774d2ca551"
Oct 30 07:20:15.759925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.759877    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:15.760091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.759979    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:15.760841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:15.760787    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:16.781523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:16.774535    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerStarted","Data":"13c5e003e549a5147986e76a783d04758eb1359ac5dbe63dc6c5a4588fa7efc6"}
Oct 30 07:20:17.026979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:17.026939    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:20:17.027357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:17.027331    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:17.027484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:17.027390    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:17.218534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:17.218495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:18.027042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:18.027005    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:18.027537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:18.027065    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:18.494208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:18.494169    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:19.026500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.026466    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:19.026708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.026528    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:19.333808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.333710    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:19.568319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.565185    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:19.639676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.639636    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:20:19.660318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:19.657491    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:20:20.027033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.026553    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:20.027033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.026630    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:20.632202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.632133    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:20.640763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.640739    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:20:20.664726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664696    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.664897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664742    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.664897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664772    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.664897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664821    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.664897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664843    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.665090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.664899    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.665090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665001    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.665090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665030    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.665266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665097    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.665266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665169    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\") pod \"2f007afb-cf33-42ae-9ffb-ef69e0504513\" (UID: \"2f007afb-cf33-42ae-9ffb-ef69e0504513\") "
Oct 30 07:20:20.665266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665175    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.665266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665206    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:20.665483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665368    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.665522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665503    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.665792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665740    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.665792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665763    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.665792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665779    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.665792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665791    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f007afb-cf33-42ae-9ffb-ef69e0504513-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.666009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.665803    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.667272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.667247    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.667447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.667414    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "kube-api-access-982rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:20.676573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.676443    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:20.676573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.676527    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo" pod "2f007afb-cf33-42ae-9ffb-ef69e0504513" (UID: "2f007afb-cf33-42ae-9ffb-ef69e0504513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:20.753324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.751924    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:20.767745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.767656    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.767745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.767690    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.767745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.767705    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982rb\" (UniqueName: \"kubernetes.io/projected/2f007afb-cf33-42ae-9ffb-ef69e0504513-kube-api-access-982rb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.767745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.767720    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f007afb-cf33-42ae-9ffb-ef69e0504513-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.863844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.863811    6738 generic.go:356] "Generic (PLEG): container finished" podID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerID="b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763" exitCode=0
Oct 30 07:20:20.864045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.863844    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" event={"ID":"2f007afb-cf33-42ae-9ffb-ef69e0504513","Type":"ContainerDied","Data":"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"}
Oct 30 07:20:20.864045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.863862    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"
Oct 30 07:20:20.864045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.863902    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z" event={"ID":"2f007afb-cf33-42ae-9ffb-ef69e0504513","Type":"ContainerDied","Data":"0b6e84a479b46f7c94a15583b6c0c2040af4f86ad581a03b1d33c86d3475588a"}
Oct 30 07:20:20.864045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.863923    6738 scope.go:122] "RemoveContainer" containerID="b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"
Oct 30 07:20:20.921801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.920923    6738 scope.go:122] "RemoveContainer" containerID="b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"
Oct 30 07:20:20.924808 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:20.924705    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763\": container with ID starting with b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763 not found: ID does not exist" containerID="b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"
Oct 30 07:20:20.924808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.924753    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763"} err="failed to get container status \"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763\": rpc error: code = NotFound desc = could not find container \"b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763\": container with ID starting with b80bcc0b8915eb28437b34fdc6fe14e9430a40d5710367f7d4640166b1ca5763 not found: ID does not exist"
Oct 30 07:20:20.940673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.940023    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:20:20.940673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:20.940268    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" containerID="cri-o://1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734" gracePeriod=30
Oct 30 07:20:21.004642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.004599    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:21.026677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.026639    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:21.026863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.026694    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:21.039672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.039627    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:20:21.045522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.045484    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-np76z"]
Oct 30 07:20:21.546547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.546509    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"]
Oct 30 07:20:21.547172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547156    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy"
Oct 30 07:20:21.547219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547174    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy"
Oct 30 07:20:21.547219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547210    6738 cpu_manager.go:414] "containerMap: removing container" podUID="307eb1bf-09ed-4d5d-adc6-005e994af3de" containerName="payload-processing"
Oct 30 07:20:21.547219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547215    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307eb1bf-09ed-4d5d-adc6-005e994af3de" containerName="payload-processing"
Oct 30 07:20:21.547347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547336    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" containerName="istio-proxy"
Oct 30 07:20:21.547393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.547352    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="307eb1bf-09ed-4d5d-adc6-005e994af3de" containerName="payload-processing"
Oct 30 07:20:21.550607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.550587    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.566441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.566283    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:20:21.566441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.566412    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:20:21.569935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.569903    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"]
Oct 30 07:20:21.621265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.621231    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:20:21.621482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.621431    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:20:21.702882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.702847    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f007afb-cf33-42ae-9ffb-ef69e0504513" path="/var/lib/kubelet/pods/2f007afb-cf33-42ae-9ffb-ef69e0504513/volumes"
Oct 30 07:20:21.712593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.712560    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.712760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.712723    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jqf\" (UniqueName: \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.712828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.712780    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.813745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.813645    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jqf\" (UniqueName: \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.813745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.813723    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.813999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.813844    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.814624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.814592    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.816143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.816110    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.837468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.837436    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jqf\" (UniqueName: \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:21.864681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:21.864653    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:22.026997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.026954    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 07:20:22.027185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.027029    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 07:20:22.142945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.142911    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:22.189121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.188966    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"]
Oct 30 07:20:22.227658 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:22.227609    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fa949c_5407_4a5f_8997_86364936d28c.slice/crio-52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7 WatchSource:0}: Error finding container 52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7: Status 404 returned error can't find the container with id 52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7
Oct 30 07:20:22.884325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.884282    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs" event={"ID":"99fa949c-5407-4a5f-8997-86364936d28c","Type":"ContainerStarted","Data":"b39afb3fcfeea1d046848b077be3355ca05e440e67d2ad61d4cb421a5fb56479"}
Oct 30 07:20:22.884843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.884351    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs" event={"ID":"99fa949c-5407-4a5f-8997-86364936d28c","Type":"ContainerStarted","Data":"52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7"}
Oct 30 07:20:22.926837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:22.926777    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs" podStartSLOduration=1.926764333 podStartE2EDuration="1.926764333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:22.926381866 +0000 UTC m=+1383.359145762" watchObservedRunningTime="2026-10-30 07:20:22.926764333 +0000 UTC m=+1383.359528225"
Oct 30 07:20:23.039796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.039754    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:20:23.040226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.040202    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:20:23.041218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.041193    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:20:23.205464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.205428    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:23.219666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.219632    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:23.894032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.894000    6738 generic.go:356] "Generic (PLEG): container finished" podID="99fa949c-5407-4a5f-8997-86364936d28c" containerID="b39afb3fcfeea1d046848b077be3355ca05e440e67d2ad61d4cb421a5fb56479" exitCode=0
Oct 30 07:20:23.894503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.894089    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs" event={"ID":"99fa949c-5407-4a5f-8997-86364936d28c","Type":"ContainerDied","Data":"b39afb3fcfeea1d046848b077be3355ca05e440e67d2ad61d4cb421a5fb56479"}
Oct 30 07:20:23.905574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.905546    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:20:23.926445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.926407    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:20:23.926626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:23.926465    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:20:24.071117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.071078    6738 scope.go:122] "RemoveContainer" containerID="ce4db065c70bd81dc4deed9ad17bab8e26ac0d798b13cb329fb043924ad24844"
Oct 30 07:20:24.081479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.081454    6738 scope.go:122] "RemoveContainer" containerID="3fb3545e8d8eb3c6993881fffbeb1c4482b8c2f5eb6079d3e213e3dcb883d1ed"
Oct 30 07:20:24.098548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.093370    6738 scope.go:122] "RemoveContainer" containerID="ea960f533916189e13e995ba476e30d25245e4310e60419faa9e13a286c2933c"
Oct 30 07:20:24.112044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.111911    6738 scope.go:122] "RemoveContainer" containerID="07608fcca7d0e4da471392b80ed000c5ddd3a4d027db2e05d8e49bc9ccd5e278"
Oct 30 07:20:24.123828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.123801    6738 scope.go:122] "RemoveContainer" containerID="b43b463f2c3adc3b31d3dcfc80a3a9ef8975bd6d235b8ce73f8d38c8d9fc5c8d"
Oct 30 07:20:24.332432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:24.332398    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:25.064724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.064696    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:25.173204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.173121    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\") pod \"99fa949c-5407-4a5f-8997-86364936d28c\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") "
Oct 30 07:20:25.173434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.173216    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\" (UniqueName: \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\") pod \"99fa949c-5407-4a5f-8997-86364936d28c\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") "
Oct 30 07:20:25.173434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.173409    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\") pod \"99fa949c-5407-4a5f-8997-86364936d28c\" (UID: \"99fa949c-5407-4a5f-8997-86364936d28c\") "
Oct 30 07:20:25.173626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.173599    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca" pod "99fa949c-5407-4a5f-8997-86364936d28c" (UID: "99fa949c-5407-4a5f-8997-86364936d28c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:25.173855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.173837    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:25.175503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.175476    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf" pod "99fa949c-5407-4a5f-8997-86364936d28c" (UID: "99fa949c-5407-4a5f-8997-86364936d28c"). InnerVolumeSpecName "kube-api-access-z6jqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:25.175630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.175561    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca" pod "99fa949c-5407-4a5f-8997-86364936d28c" (UID: "99fa949c-5407-4a5f-8997-86364936d28c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:20:25.275241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.275200    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa949c-5407-4a5f-8997-86364936d28c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:25.275241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.275233    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jqf\" (UniqueName: \"kubernetes.io/projected/99fa949c-5407-4a5f-8997-86364936d28c-kube-api-access-z6jqf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:25.484262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.484180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:25.911964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.911930    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"
Oct 30 07:20:25.912154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.911968    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs" event={"ID":"99fa949c-5407-4a5f-8997-86364936d28c","Type":"ContainerDied","Data":"52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7"}
Oct 30 07:20:25.912154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:25.912016    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52e16180f25e97407d4b17b0661b852477f4c2fa14aa1e8913ebb73cb479eab7"
Oct 30 07:20:26.100746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.100707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:26.297884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.297858    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:20:26.300756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.300716    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:26.391615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391573    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.391798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391659    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.391798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391786    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.391932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391824    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.391932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391865    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.392050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391942    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:26.392050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.391953    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.392161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392057    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.392161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392094    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.392161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392125    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:26.392367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392168    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\") pod \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\" (UID: \"8d6c24f1-a0b3-466d-954e-2db6d985cf02\") "
Oct 30 07:20:26.392893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392649    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:26.392893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392669    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:26.393098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392910    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.393098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392932    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.393098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392945    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.393098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.392960    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.393436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.393400    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:26.394641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.394612    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:26.394755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.394696    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:26.395129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.395105    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:26.395391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.395365    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8" pod "8d6c24f1-a0b3-466d-954e-2db6d985cf02" (UID: "8d6c24f1-a0b3-466d-954e-2db6d985cf02"). InnerVolumeSpecName "kube-api-access-hbzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:26.422287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.422256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:26.494582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.494495    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.494582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.494523    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.494582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.494533    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.494582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.494541    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6c24f1-a0b3-466d-954e-2db6d985cf02-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.494582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.494549    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzd8\" (UniqueName: \"kubernetes.io/projected/8d6c24f1-a0b3-466d-954e-2db6d985cf02-kube-api-access-hbzd8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:26.925934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.925897    6738 generic.go:356] "Generic (PLEG): container finished" podID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerID="1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734" exitCode=0
Oct 30 07:20:26.926416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.926384    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"
Oct 30 07:20:26.926578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.926466    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerDied","Data":"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"}
Oct 30 07:20:26.926578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.926572    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b" event={"ID":"8d6c24f1-a0b3-466d-954e-2db6d985cf02","Type":"ContainerDied","Data":"520d4bf5f1f3e6d4fdc960069a7a5b84f110f5d361e6165ef4ff1c65b741aef1"}
Oct 30 07:20:26.926680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.926613    6738 scope.go:122] "RemoveContainer" containerID="1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"
Oct 30 07:20:26.938523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.938502    6738 scope.go:122] "RemoveContainer" containerID="e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"
Oct 30 07:20:26.948340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.948318    6738 scope.go:122] "RemoveContainer" containerID="1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"
Oct 30 07:20:26.948643 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:26.948621    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734\": container with ID starting with 1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734 not found: ID does not exist" containerID="1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"
Oct 30 07:20:26.948690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.948651    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734"} err="failed to get container status \"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734\": rpc error: code = NotFound desc = could not find container \"1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734\": container with ID starting with 1a62a7d72fa2383a0b47b315e960fe77d695a2550ce55972b568ca8a74342734 not found: ID does not exist"
Oct 30 07:20:26.948690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.948668    6738 scope.go:122] "RemoveContainer" containerID="e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"
Oct 30 07:20:26.948893 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:26.948873    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848\": container with ID starting with e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848 not found: ID does not exist" containerID="e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"
Oct 30 07:20:26.948934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.948897    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848"} err="failed to get container status \"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848\": rpc error: code = NotFound desc = could not find container \"e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848\": container with ID starting with e0de33dd28d19a1c03337d2de180d42d714d55b040c530a2cae3ed92524db848 not found: ID does not exist"
Oct 30 07:20:26.999970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:26.997682    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:20:27.022442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:27.022391    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-137ec6-openshift-default-69b77bbd5-lsf7b"]
Oct 30 07:20:27.690675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:27.690640    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" path="/var/lib/kubelet/pods/8d6c24f1-a0b3-466d-954e-2db6d985cf02/volumes"
Oct 30 07:20:28.212135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:28.212096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:29.932107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:29.932069    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:30.045455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:30.045407    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:30.436512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:30.436475    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.052131    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053123    6738 cpu_manager.go:414] "containerMap: removing container" podUID="99fa949c-5407-4a5f-8997-86364936d28c" containerName="revoke-keys"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053146    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fa949c-5407-4a5f-8997-86364936d28c" containerName="revoke-keys"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053170    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053178    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053219    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053227    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053520    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053550    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6c24f1-a0b3-466d-954e-2db6d985cf02" containerName="istio-proxy"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.053572    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fa949c-5407-4a5f-8997-86364936d28c" containerName="revoke-keys"
Oct 30 07:20:31.067326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.059852    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.140330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.137619    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8d44bf-openshift-default-dockercfg-52xtn\""
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162581    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162631    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162676    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162742    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162764    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zx9t\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.162875    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.163001    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.165712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.165556    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:20:31.166724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.166686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:31.264656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264604    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.264852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264671    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.264852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264703    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.264852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264745    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.264852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264786    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264859    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264883    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zx9t\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.264994    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.265104    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.265116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.265221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.265427    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.265953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.265928    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.266416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.266382    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.268083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.268014    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.271911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.271887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.288336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.288266    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.288596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.288573    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zx9t\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\") pod \"e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") " pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.356512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.356420    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:31.488434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.488398    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:31.810098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.810065    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:20:31.811086 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:31.811057    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a24618_7125_4e72_9da1_73900e657d6c.slice/crio-acc85dd18218d69594fee2cc03fbb7fbed582dc56bce4029c6bda8e7218c554e WatchSource:0}: Error finding container acc85dd18218d69594fee2cc03fbb7fbed582dc56bce4029c6bda8e7218c554e: Status 404 returned error can't find the container with id acc85dd18218d69594fee2cc03fbb7fbed582dc56bce4029c6bda8e7218c554e
Oct 30 07:20:31.814619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.814576    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:31.814706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.814668    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:31.814744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.814714    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:31.971979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.971945    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerStarted","Data":"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"}
Oct 30 07:20:31.972163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:31.972002    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerStarted","Data":"acc85dd18218d69594fee2cc03fbb7fbed582dc56bce4029c6bda8e7218c554e"}
Oct 30 07:20:32.019490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.019437    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podStartSLOduration=1.019420638 podStartE2EDuration="1.019420638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:32.017538959 +0000 UTC m=+1392.450302847" watchObservedRunningTime="2026-10-30 07:20:32.019420638 +0000 UTC m=+1392.452184530"
Oct 30 07:20:32.130468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.130430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:32.489085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.489000    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:32.494937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.494910    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:32.978912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.978882    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:32.979800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:32.979779    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:20:33.845194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:33.845155    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:33.935989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:33.935946    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:20:33.936211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:33.936186    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" podUID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" containerName="payload-processing" containerID="cri-o://031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2" gracePeriod=30
Oct 30 07:20:33.962003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:33.961967    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:20:33.962222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:33.962199    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerName="payload-pre-processing" containerID="cri-o://17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa" gracePeriod=30
Oct 30 07:20:34.028875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.028832    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:34.052160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.052118    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.181:9004: connect: connection refused"
Oct 30 07:20:34.151108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.151073    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:34.301839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.301811    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:20:34.305938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.305915    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:20:34.428930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.428807    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\" (UniqueName: \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\") pod \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") "
Oct 30 07:20:34.481270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.439446    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\") pod \"c33ca108-63a1-4c34-88c9-2d6951722ab2\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") "
Oct 30 07:20:34.481270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.439646    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\" (UniqueName: \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\") pod \"c33ca108-63a1-4c34-88c9-2d6951722ab2\" (UID: \"c33ca108-63a1-4c34-88c9-2d6951722ab2\") "
Oct 30 07:20:34.481270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.439753    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\") pod \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\" (UID: \"1dcdfd68-aaa5-4187-bdc2-2b7558f81351\") "
Oct 30 07:20:34.481270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.454903    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume" pod "c33ca108-63a1-4c34-88c9-2d6951722ab2" (UID: "c33ca108-63a1-4c34-88c9-2d6951722ab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:34.526493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.493575    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume" pod "1dcdfd68-aaa5-4187-bdc2-2b7558f81351" (UID: "1dcdfd68-aaa5-4187-bdc2-2b7558f81351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:34.526771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.521567    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2" pod "1dcdfd68-aaa5-4187-bdc2-2b7558f81351" (UID: "1dcdfd68-aaa5-4187-bdc2-2b7558f81351"). InnerVolumeSpecName "kube-api-access-lxbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:34.526944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.523641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6" pod "c33ca108-63a1-4c34-88c9-2d6951722ab2" (UID: "c33ca108-63a1-4c34-88c9-2d6951722ab2"). InnerVolumeSpecName "kube-api-access-nj8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:34.542326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.542272    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8f6\" (UniqueName: \"kubernetes.io/projected/c33ca108-63a1-4c34-88c9-2d6951722ab2-kube-api-access-nj8f6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:34.542326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.542327    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:34.542625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.542343    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbk2\" (UniqueName: \"kubernetes.io/projected/1dcdfd68-aaa5-4187-bdc2-2b7558f81351-kube-api-access-lxbk2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:34.542625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.542358    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33ca108-63a1-4c34-88c9-2d6951722ab2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:34.559145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.559095    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:34.715492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.715395    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:34.755115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.755084    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"]
Oct 30 07:20:34.759369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.759340    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7467d-fk2fs"]
Oct 30 07:20:34.827367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.827286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:34.957477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:34.957441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:35.007095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.007008    6738 generic.go:356] "Generic (PLEG): container finished" podID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" containerID="031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2" exitCode=0
Oct 30 07:20:35.007095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.007054    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"
Oct 30 07:20:35.007378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.007095    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" event={"ID":"1dcdfd68-aaa5-4187-bdc2-2b7558f81351","Type":"ContainerDied","Data":"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"}
Oct 30 07:20:35.007378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.007149    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97" event={"ID":"1dcdfd68-aaa5-4187-bdc2-2b7558f81351","Type":"ContainerDied","Data":"22b2a8331cd06945402cf065d5c39d0e214ef2152f6d271bfeccd79564f761a6"}
Oct 30 07:20:35.007378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.007178    6738 scope.go:122] "RemoveContainer" containerID="031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"
Oct 30 07:20:35.008710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.008688    6738 generic.go:356] "Generic (PLEG): container finished" podID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerID="17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa" exitCode=0
Oct 30 07:20:35.008823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.008738    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"
Oct 30 07:20:35.008823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.008776    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" event={"ID":"c33ca108-63a1-4c34-88c9-2d6951722ab2","Type":"ContainerDied","Data":"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"}
Oct 30 07:20:35.008918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.008828    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l" event={"ID":"c33ca108-63a1-4c34-88c9-2d6951722ab2","Type":"ContainerDied","Data":"ffb0b6c0ace2387c93ebe999bd92a6030a2a2d63b165caff2fabeb0c2d9fbfe8"}
Oct 30 07:20:35.020983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.020960    6738 scope.go:122] "RemoveContainer" containerID="031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"
Oct 30 07:20:35.021375 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:35.021351    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2\": container with ID starting with 031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2 not found: ID does not exist" containerID="031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"
Oct 30 07:20:35.021445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.021383    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2"} err="failed to get container status \"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2\": rpc error: code = NotFound desc = could not find container \"031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2\": container with ID starting with 031d635c892674273c740173baf1bcc706ba4774f6472af4f6690ffdd5d105b2 not found: ID does not exist"
Oct 30 07:20:35.021445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.021404    6738 scope.go:122] "RemoveContainer" containerID="17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"
Oct 30 07:20:35.031918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.031899    6738 scope.go:122] "RemoveContainer" containerID="17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"
Oct 30 07:20:35.032246 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:35.032225    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa\": container with ID starting with 17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa not found: ID does not exist" containerID="17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"
Oct 30 07:20:35.032356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.032253    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa"} err="failed to get container status \"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa\": rpc error: code = NotFound desc = could not find container \"17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa\": container with ID starting with 17e2d5e38fe7f2274d0a2afb04c0601d7ea3b831fc8a1b7070a2f6a4d050b2fa not found: ID does not exist"
Oct 30 07:20:35.070476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.070442    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:20:35.082854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.082821    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7467d-5fd94cbcd4-4bs97"]
Oct 30 07:20:35.110691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.110656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:35.119898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.119864    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:20:35.124871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.124835    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7467d-69b77896f9-fvq2l"]
Oct 30 07:20:35.279914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.279824    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:35.691057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.691025    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" path="/var/lib/kubelet/pods/1dcdfd68-aaa5-4187-bdc2-2b7558f81351/volumes"
Oct 30 07:20:35.691515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.691502    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fa949c-5407-4a5f-8997-86364936d28c" path="/var/lib/kubelet/pods/99fa949c-5407-4a5f-8997-86364936d28c/volumes"
Oct 30 07:20:35.691857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:35.691845    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" path="/var/lib/kubelet/pods/c33ca108-63a1-4c34-88c9-2d6951722ab2/volumes"
Oct 30 07:20:36.879617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.879580    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:20:36.880376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880355    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerName="payload-pre-processing"
Oct 30 07:20:36.880476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880377    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerName="payload-pre-processing"
Oct 30 07:20:36.880476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880422    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" containerName="payload-processing"
Oct 30 07:20:36.880476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880430    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" containerName="payload-processing"
Oct 30 07:20:36.880660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880624    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33ca108-63a1-4c34-88c9-2d6951722ab2" containerName="payload-pre-processing"
Oct 30 07:20:36.880660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.880644    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcdfd68-aaa5-4187-bdc2-2b7558f81351" containerName="payload-processing"
Oct 30 07:20:36.882970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.882949    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:36.901183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.901155    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:20:36.915339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.910050    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8d44bf-dockercfg-krj64\""
Oct 30 07:20:36.915339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:36.910120    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8d44bf\""
Oct 30 07:20:37.066364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.048578    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.066364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.048711    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qwq\" (UniqueName: \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.086325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.078745    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:20:37.090319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.086541    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:20:37.090319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.086696    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.150277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.150241    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.150512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.150399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qwq\" (UniqueName: \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.151202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.151179    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.174680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.174649    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qwq\" (UniqueName: \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.195527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.195492    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:37.251587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.251392    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.251587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.251448    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4rc\" (UniqueName: \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.352753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.352713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.352753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.352757    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4rc\" (UniqueName: \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.353511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.353488    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.378718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.378683    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4rc\" (UniqueName: \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\") pod \"payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.445330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.445231    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:37.504029 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:37.503747    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74982c3c_b845_44f9_937c_e2e7566b7a73.slice/crio-e33f04a2514d42791cd8eda2678917fbf694824b0769e1d80911e891f5de9ff6 WatchSource:0}: Error finding container e33f04a2514d42791cd8eda2678917fbf694824b0769e1d80911e891f5de9ff6: Status 404 returned error can't find the container with id e33f04a2514d42791cd8eda2678917fbf694824b0769e1d80911e891f5de9ff6
Oct 30 07:20:37.520159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.509399    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:20:37.537584 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:37.537546    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:20:37.751660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.751635    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:20:37.753432 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:37.753401    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288bcd04_57c1_4766_a648_f9673e990f02.slice/crio-f1a709586906dfa7090ffd3e8c617d3a5cbba6126bba98d4deab85ef8ca83e04 WatchSource:0}: Error finding container f1a709586906dfa7090ffd3e8c617d3a5cbba6126bba98d4deab85ef8ca83e04: Status 404 returned error can't find the container with id f1a709586906dfa7090ffd3e8c617d3a5cbba6126bba98d4deab85ef8ca83e04
Oct 30 07:20:37.976992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:37.976949    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:38.079276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.079150    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" event={"ID":"288bcd04-57c1-4766-a648-f9673e990f02","Type":"ContainerStarted","Data":"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"}
Oct 30 07:20:38.079276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.079252    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" event={"ID":"288bcd04-57c1-4766-a648-f9673e990f02","Type":"ContainerStarted","Data":"f1a709586906dfa7090ffd3e8c617d3a5cbba6126bba98d4deab85ef8ca83e04"}
Oct 30 07:20:38.079549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.079325    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:38.080924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.080894    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" event={"ID":"74982c3c-b845-44f9-937c-e2e7566b7a73","Type":"ContainerStarted","Data":"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"}
Oct 30 07:20:38.081029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.080944    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" event={"ID":"74982c3c-b845-44f9-937c-e2e7566b7a73","Type":"ContainerStarted","Data":"e33f04a2514d42791cd8eda2678917fbf694824b0769e1d80911e891f5de9ff6"}
Oct 30 07:20:38.081029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.080967    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:38.124451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.124403    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" podStartSLOduration=2.124389345 podStartE2EDuration="2.124389345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:38.121557362 +0000 UTC m=+1398.554321271" watchObservedRunningTime="2026-10-30 07:20:38.124389345 +0000 UTC m=+1398.557153238"
Oct 30 07:20:38.124909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.124882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:38.168698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.168639    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" podStartSLOduration=2.168624043 podStartE2EDuration="2.168624043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:38.165554257 +0000 UTC m=+1398.598318149" watchObservedRunningTime="2026-10-30 07:20:38.168624043 +0000 UTC m=+1398.601387933"
Oct 30 07:20:38.669073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:38.669043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:40.595784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:40.595684    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:41.574712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.574672    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:41.690125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.688989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:41.807667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.807623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:41.910178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.910141    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:20:41.910905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.910869    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy" containerID="cri-o://b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921" gracePeriod=30
Oct 30 07:20:41.938502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:41.938464    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:42.061706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:42.061664    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:43.055054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:43.055001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:43.574455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:43.574401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:44.012497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.012460    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:20:44.017629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.017599    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.047453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.047416    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:20:44.055707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.055675    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2504c2-openshift-default-dockercfg-rmk4j\""
Oct 30 07:20:44.067088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067053    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067100    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067241    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067269    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067374    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067400    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067446    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz9g\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067512    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.067713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.067578    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.107445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.107407    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195261    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195342    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195410    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195439    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195486    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz9g\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195553    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195618    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195702    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.195743    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.198319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.196254    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.224321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.202969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.224321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.203898    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.224321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.204160    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.224321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.208532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.250316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.237308    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.250316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.249548    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.290401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.289873    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.323506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.323426    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz9g\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\") pod \"e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") " pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.336611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.336567    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:44.447924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.447892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:44.712957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.712921    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:20:44.715597 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:44.715565    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb88b431_45a8_43cd_b43f_5feb2b706559.slice/crio-830dc10a60bc17db9f29d6d4ad5fe8b42c224b9142ac731f82bd601053efac78 WatchSource:0}: Error finding container 830dc10a60bc17db9f29d6d4ad5fe8b42c224b9142ac731f82bd601053efac78: Status 404 returned error can't find the container with id 830dc10a60bc17db9f29d6d4ad5fe8b42c224b9142ac731f82bd601053efac78
Oct 30 07:20:44.718923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.718887    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:44.719001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.718962    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:44.719001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:44.718993    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:45.198366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.198219    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:45.332226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.328790    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerStarted","Data":"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"}
Oct 30 07:20:45.332226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.328862    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerStarted","Data":"830dc10a60bc17db9f29d6d4ad5fe8b42c224b9142ac731f82bd601053efac78"}
Oct 30 07:20:45.337548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.337507    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:45.343030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.343006    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:45.434985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.434906    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podStartSLOduration=2.434889776 podStartE2EDuration="2.434889776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:45.384950647 +0000 UTC m=+1405.817714540" watchObservedRunningTime="2026-10-30 07:20:45.434889776 +0000 UTC m=+1405.867653733"
Oct 30 07:20:45.603449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.603350    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:45.721209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:45.721171    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:46.336211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:46.336169    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:46.337397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:46.337376    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:20:47.225824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.225799    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:20:47.263393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263354    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263410    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263452    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263482    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263529    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263576    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263599    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263667    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.263952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263730    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\") pod \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\" (UID: \"4e77782b-d1c7-4b84-80e4-11b2e92d07a0\") "
Oct 30 07:20:47.264433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263706    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.264433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263755    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.264433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263889    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:47.264433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.263886    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.264433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264156    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.264708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264475    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.264708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264499    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.264708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264514    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.264708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264527    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.264708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.264542    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.265829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.265801    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.265955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.265925    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:47.266027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.265999    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "kube-api-access-x5f5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:47.266180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.266165    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo" pod "4e77782b-d1c7-4b84-80e4-11b2e92d07a0" (UID: "4e77782b-d1c7-4b84-80e4-11b2e92d07a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:47.339008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.338949    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:47.346554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.346520    6738 generic.go:356] "Generic (PLEG): container finished" podID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerID="b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921" exitCode=0
Oct 30 07:20:47.346760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.346570    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"
Oct 30 07:20:47.346760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.346612    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerDied","Data":"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"}
Oct 30 07:20:47.346760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.346674    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx" event={"ID":"4e77782b-d1c7-4b84-80e4-11b2e92d07a0","Type":"ContainerDied","Data":"41cf767dade6d295a4194147ebd5064e252bf335a67c11f0a81299ff75fa7726"}
Oct 30 07:20:47.346760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.346707    6738 scope.go:122] "RemoveContainer" containerID="b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"
Oct 30 07:20:47.398750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.366016    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.398750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.366046    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.398750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.366059    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.398750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.366077    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5f5s\" (UniqueName: \"kubernetes.io/projected/4e77782b-d1c7-4b84-80e4-11b2e92d07a0-kube-api-access-x5f5s\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.473511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.466152    6738 scope.go:122] "RemoveContainer" containerID="115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"
Oct 30 07:20:47.503319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.498481    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:20:47.503319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.501765    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7467d-openshift-default-6d9ffd6585-wd8lx"]
Oct 30 07:20:47.511820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.511802    6738 scope.go:122] "RemoveContainer" containerID="b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"
Oct 30 07:20:47.512371 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:47.512186    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921\": container with ID starting with b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921 not found: ID does not exist" containerID="b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"
Oct 30 07:20:47.512371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.512217    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921"} err="failed to get container status \"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921\": rpc error: code = NotFound desc = could not find container \"b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921\": container with ID starting with b876c2f37ece0c3104f02dc409399a5a41edb2938cc63de40c02e82ac7a84921 not found: ID does not exist"
Oct 30 07:20:47.512371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.512237    6738 scope.go:122] "RemoveContainer" containerID="115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"
Oct 30 07:20:47.512586 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:20:47.512566    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3\": container with ID starting with 115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3 not found: ID does not exist" containerID="115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"
Oct 30 07:20:47.512629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.512595    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3"} err="failed to get container status \"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3\": rpc error: code = NotFound desc = could not find container \"115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3\": container with ID starting with 115aa0fdc0d7db1cd7678d3e207ace38a123bf4501c9382aceb5ea189956dac3 not found: ID does not exist"
Oct 30 07:20:47.548661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.548629    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"]
Oct 30 07:20:47.549267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.549252    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:47.549337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.549268    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:47.549337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.549282    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:47.549337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.549287    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:47.549463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.549402    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:47.553058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.553039    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.602602    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.602692    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.602737    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.602828    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.606364    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"]
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.606537    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.606620    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l42n\" (UniqueName: \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.607355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.606791    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.704332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.704285    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" path="/var/lib/kubelet/pods/4e77782b-d1c7-4b84-80e4-11b2e92d07a0/volumes"
Oct 30 07:20:47.708559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.708527    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.708750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.708643    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.708750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.708735    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l42n\" (UniqueName: \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.709383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.709364    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.710985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.710962    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.732381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.732278    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l42n\" (UniqueName: \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\") pod \"maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.814276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.814233    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:47.877727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.876602    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:47.936466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:47.936031    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:48.199943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:48.199908    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"]
Oct 30 07:20:48.200770 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:48.200744    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9267eb_8e18_4113_9363_15ce62068d1c.slice/crio-c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06 WatchSource:0}: Error finding container c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06: Status 404 returned error can't find the container with id c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06
Oct 30 07:20:48.356191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:48.356153    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr" event={"ID":"3d9267eb-8e18-4113-9363-15ce62068d1c","Type":"ContainerStarted","Data":"c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06"}
Oct 30 07:20:49.104149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:49.103259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:20:49.106384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:49.104696    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:20:49.366318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:49.366215    6738 generic.go:356] "Generic (PLEG): container finished" podID="3d9267eb-8e18-4113-9363-15ce62068d1c" containerID="3d50c3ee5f52d665fddab44037f722d15c88fb5e3e19c55c3891a655a865c256" exitCode=0
Oct 30 07:20:49.366709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:49.366323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr" event={"ID":"3d9267eb-8e18-4113-9363-15ce62068d1c","Type":"ContainerDied","Data":"3d50c3ee5f52d665fddab44037f722d15c88fb5e3e19c55c3891a655a865c256"}
Oct 30 07:20:49.443424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:49.443387    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:50.274866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.274791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:50.416131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.416094    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:20:50.416904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.416880    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e77782b-d1c7-4b84-80e4-11b2e92d07a0" containerName="istio-proxy"
Oct 30 07:20:50.419630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.419609    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.447196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.447161    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:20:50.459126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.459084    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2504c2\""
Oct 30 07:20:50.460040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.459990    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.460209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.460129    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gfp\" (UniqueName: \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.486241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.486194    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:20:50.493410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.491315    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.532397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.531142    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2504c2-dockercfg-76dsz\""
Oct 30 07:20:50.536731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.536700    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:20:50.543223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.540989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:50.561327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.561101    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.561327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.561233    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.562541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.561381    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.562541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.561420    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gfp\" (UniqueName: \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.562777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.562750    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.580327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.579319    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:50.588513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.588477    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gfp\" (UniqueName: \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\") pod \"payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.716317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.707980    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\") pod \"3d9267eb-8e18-4113-9363-15ce62068d1c\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") "
Oct 30 07:20:50.750243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.725337    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\" (UniqueName: \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\") pod \"3d9267eb-8e18-4113-9363-15ce62068d1c\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") "
Oct 30 07:20:50.750243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.725517    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\") pod \"3d9267eb-8e18-4113-9363-15ce62068d1c\" (UID: \"3d9267eb-8e18-4113-9363-15ce62068d1c\") "
Oct 30 07:20:50.750243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.725854    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.750243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.726025    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.785008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.773592    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:50.785008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.778385    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.787511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.786391    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca" pod "3d9267eb-8e18-4113-9363-15ce62068d1c" (UID: "3d9267eb-8e18-4113-9363-15ce62068d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:50.795660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.795632    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca" pod "3d9267eb-8e18-4113-9363-15ce62068d1c" (UID: "3d9267eb-8e18-4113-9363-15ce62068d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:20:50.800685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.800481    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n" pod "3d9267eb-8e18-4113-9363-15ce62068d1c" (UID: "3d9267eb-8e18-4113-9363-15ce62068d1c"). InnerVolumeSpecName "kube-api-access-5l42n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:50.809550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.809455    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\") pod \"payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.818171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.818136    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:50.828244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.827991    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:50.828244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.828023    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l42n\" (UniqueName: \"kubernetes.io/projected/3d9267eb-8e18-4113-9363-15ce62068d1c-kube-api-access-5l42n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:50.828244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:50.828038    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9267eb-8e18-4113-9363-15ce62068d1c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:20:51.109193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.109164    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:20:51.163094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.163068    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:20:51.386069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.386043    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"
Oct 30 07:20:51.386270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.386046    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr" event={"ID":"3d9267eb-8e18-4113-9363-15ce62068d1c","Type":"ContainerDied","Data":"c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06"}
Oct 30 07:20:51.386270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.386187    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6c523c5ecd90b3a9b10133d8ffc6715b1ea170e9d4cf452cb91f95872e00f06"
Oct 30 07:20:51.388016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.387990    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" event={"ID":"3f69fd1e-567c-468c-af03-a7f5fdf84e4a","Type":"ContainerStarted","Data":"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"}
Oct 30 07:20:51.388147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.388033    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" event={"ID":"3f69fd1e-567c-468c-af03-a7f5fdf84e4a","Type":"ContainerStarted","Data":"d3d851bf2565232555234695bdd96f12d7e970731fbb0c10c3b4338fcc094d1c"}
Oct 30 07:20:51.388147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.388068    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:20:51.390066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.390038    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" event={"ID":"87305586-cd7e-4814-bd76-09c27728dafc","Type":"ContainerStarted","Data":"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"}
Oct 30 07:20:51.390216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.390084    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" event={"ID":"87305586-cd7e-4814-bd76-09c27728dafc","Type":"ContainerStarted","Data":"795cf1bc17661f5766b06aeb886f8f5cf97a8c212f39a9e0cc8b6c1cc145f115"}
Oct 30 07:20:51.390216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.390177    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:20:51.431734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.431685    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" podStartSLOduration=1.431670984 podStartE2EDuration="1.431670984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:51.428885609 +0000 UTC m=+1411.861649503" watchObservedRunningTime="2026-10-30 07:20:51.431670984 +0000 UTC m=+1411.864434877"
Oct 30 07:20:51.471526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.471469    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" podStartSLOduration=1.471454049 podStartE2EDuration="1.471454049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:51.469207472 +0000 UTC m=+1411.901971365" watchObservedRunningTime="2026-10-30 07:20:51.471454049 +0000 UTC m=+1411.904217943"
Oct 30 07:20:51.634445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:51.634405    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:53.406963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.406871    6738 generic.go:356] "Generic (PLEG): container finished" podID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerID="94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f" exitCode=0
Oct 30 07:20:53.407467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.406951    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerDied","Data":"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"}
Oct 30 07:20:53.407467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.407412    6738 scope.go:122] "RemoveContainer" containerID="94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"
Oct 30 07:20:53.409174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.409147    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:53.409249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.409219    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:53.409284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.409253    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:53.545614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:53.545583    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:54.416777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:54.416744    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerStarted","Data":"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"}
Oct 30 07:20:54.549941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:54.549896    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:54.550203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:54.550180    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 07:20:54.550265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:54.550240    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 07:20:54.940880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:54.940829    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:55.405007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:55.404972    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:55.519481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:55.519440    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:55.563627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:55.563593    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 07:20:55.567611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:55.567533    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 07:20:55.647446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:55.647404    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:56.239076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.239032    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:56.549498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.549415    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 07:20:56.549498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.549485    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 07:20:56.611496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.611455    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:20:56.612450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.612429    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3d9267eb-8e18-4113-9363-15ce62068d1c" containerName="revoke-keys"
Oct 30 07:20:56.612511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.612454    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9267eb-8e18-4113-9363-15ce62068d1c" containerName="revoke-keys"
Oct 30 07:20:56.612688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.612675    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9267eb-8e18-4113-9363-15ce62068d1c" containerName="revoke-keys"
Oct 30 07:20:56.615665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.615649    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.630791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.630762    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-704bf6-openshift-default-dockercfg-dtdjx\""
Oct 30 07:20:56.648503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.648462    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:20:56.692955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.692909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:56.735180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735140    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735207    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735274    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735387    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735495    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735566    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtmg\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735742    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735818    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.735928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.735891    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837139    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtmg\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837272    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837387    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837442    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837478    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837525    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837610    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837753    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.837947    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.844370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.843758    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.858316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.845240    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.858316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.845799    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.858316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.846375    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.858316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.846559    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.868318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.863904    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.883314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.875924    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.883314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.876397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.922322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.920053    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtmg\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:56.930893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.930564    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:57.000049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:56.993208    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:57.395818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.395786    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:20:57.397490 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:20:57.397458    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73019e5b_4470_4369_9ada_d7e27ae79f54.slice/crio-90cc7b7808ef1447cb11f84abf5c9aabac9e7ec09c8d394611721530d53ff735 WatchSource:0}: Error finding container 90cc7b7808ef1447cb11f84abf5c9aabac9e7ec09c8d394611721530d53ff735: Status 404 returned error can't find the container with id 90cc7b7808ef1447cb11f84abf5c9aabac9e7ec09c8d394611721530d53ff735
Oct 30 07:20:57.400079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.400045    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:57.400173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.400120    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:57.400173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.400152    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:20:57.450907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.450872    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerStarted","Data":"90cc7b7808ef1447cb11f84abf5c9aabac9e7ec09c8d394611721530d53ff735"}
Oct 30 07:20:57.549967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.549811    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 07:20:57.549967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.549880    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 07:20:57.598333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:57.590825    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:58.459822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.459789    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerStarted","Data":"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"}
Oct 30 07:20:58.549777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.549741    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 07:20:58.549979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.549792    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 07:20:58.555390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.555345    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podStartSLOduration=2.555331018 podStartE2EDuration="2.555331018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:58.54027252 +0000 UTC m=+1418.973036437" watchObservedRunningTime="2026-10-30 07:20:58.555331018 +0000 UTC m=+1418.988094910"
Oct 30 07:20:58.937350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.937308    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:58.943275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:58.943243    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:59.467188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.467156    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:59.468323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.468278    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:20:59.565512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.565478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:20:59.672321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.670484    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:59.672321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.672058    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:20:59.699666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:20:59.699638    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:21:01.012527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:01.012489    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:01.153041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:01.152996    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:01.543961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:01.543928    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:02.400774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.400745    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:21:02.401215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.400810    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:21:02.731219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.730603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:02.848653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.848615    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:21:02.852483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.852456    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:02.861750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.860060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:02.867824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.867795    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-704bf6\""
Oct 30 07:21:02.867824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.867817    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-704bf6-dockercfg-6jg59\""
Oct 30 07:21:02.871574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.871544    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:21:02.887857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.887827    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:21:02.891400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.891369    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:02.913173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.913141    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:21:02.978315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.975246    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:02.978315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.975323    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2v5\" (UniqueName: \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:02.978315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.975482    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:02.978315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:02.975581    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwb5v\" (UniqueName: \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.078423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.077203    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.078423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.077253    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2v5\" (UniqueName: \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:03.078423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.077377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:03.078423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.077428    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwb5v\" (UniqueName: \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.078808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.078628    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.090203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.079186    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:03.122361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.120053    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwb5v\" (UniqueName: \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\") pod \"payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.127950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.127411    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2v5\" (UniqueName: \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\") pod \"payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:03.184619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.184118    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:03.211134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.211102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:03.212714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.212687    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:03.313061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.312963    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:21:03.313566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.313535    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" podUID="b823a536-42e3-41fc-bf1c-ee35d171efdd" containerName="payload-processing" containerID="cri-o://f3fe6d3d7006fa2401a6bc3b09f0446ec1d83be762a8828c4ccc621d42f981df" gracePeriod=30
Oct 30 07:21:03.335051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.334977    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:21:03.335209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.335174    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" podUID="37d1bdca-7123-428e-9be1-61cd72e855a2" containerName="payload-pre-processing" containerID="cri-o://e8eb16adffbf8217de59f61afacba3048cda81c3324b2f5d84786f39bd9b384e" gracePeriod=30
Oct 30 07:21:03.412072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.412036    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:03.509075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.509023    6738 generic.go:356] "Generic (PLEG): container finished" podID="b823a536-42e3-41fc-bf1c-ee35d171efdd" containerID="f3fe6d3d7006fa2401a6bc3b09f0446ec1d83be762a8828c4ccc621d42f981df" exitCode=0
Oct 30 07:21:03.509267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.509222    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" event={"ID":"b823a536-42e3-41fc-bf1c-ee35d171efdd","Type":"ContainerDied","Data":"f3fe6d3d7006fa2401a6bc3b09f0446ec1d83be762a8828c4ccc621d42f981df"}
Oct 30 07:21:03.511685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.511465    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:21:03.519155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.518383    6738 generic.go:356] "Generic (PLEG): container finished" podID="37d1bdca-7123-428e-9be1-61cd72e855a2" containerID="e8eb16adffbf8217de59f61afacba3048cda81c3324b2f5d84786f39bd9b384e" exitCode=0
Oct 30 07:21:03.519155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.518487    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" event={"ID":"37d1bdca-7123-428e-9be1-61cd72e855a2","Type":"ContainerDied","Data":"e8eb16adffbf8217de59f61afacba3048cda81c3324b2f5d84786f39bd9b384e"}
Oct 30 07:21:03.522789 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:03.522757    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105f610a_3b07_4433_807a_b851ad06e842.slice/crio-749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e WatchSource:0}: Error finding container 749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e: Status 404 returned error can't find the container with id 749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e
Oct 30 07:21:03.535715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.535663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:03.551882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.551426    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:21:03.560468 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:03.555316    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63074c69_da63_414a_8123_00065b48db97.slice/crio-3ce24cd892695168fcf8f37f7afa0c3fc4ee970cb237842b9fe0a931ee02c255 WatchSource:0}: Error finding container 3ce24cd892695168fcf8f37f7afa0c3fc4ee970cb237842b9fe0a931ee02c255: Status 404 returned error can't find the container with id 3ce24cd892695168fcf8f37f7afa0c3fc4ee970cb237842b9fe0a931ee02c255
Oct 30 07:21:03.821536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.820970    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:21:03.845024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.844275    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:21:03.916098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916060    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\") pod \"37d1bdca-7123-428e-9be1-61cd72e855a2\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") "
Oct 30 07:21:03.916362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916265    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\" (UniqueName: \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\") pod \"37d1bdca-7123-428e-9be1-61cd72e855a2\" (UID: \"37d1bdca-7123-428e-9be1-61cd72e855a2\") "
Oct 30 07:21:03.916362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916350    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\") pod \"b823a536-42e3-41fc-bf1c-ee35d171efdd\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") "
Oct 30 07:21:03.916521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916381    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\" (UniqueName: \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\") pod \"b823a536-42e3-41fc-bf1c-ee35d171efdd\" (UID: \"b823a536-42e3-41fc-bf1c-ee35d171efdd\") "
Oct 30 07:21:03.916792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916761    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume" pod "b823a536-42e3-41fc-bf1c-ee35d171efdd" (UID: "b823a536-42e3-41fc-bf1c-ee35d171efdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:03.917333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.916789    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume" pod "37d1bdca-7123-428e-9be1-61cd72e855a2" (UID: "37d1bdca-7123-428e-9be1-61cd72e855a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:03.917333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.917035    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b823a536-42e3-41fc-bf1c-ee35d171efdd-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:03.917333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.917047    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d1bdca-7123-428e-9be1-61cd72e855a2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:03.919629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.919602    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp" pod "b823a536-42e3-41fc-bf1c-ee35d171efdd" (UID: "b823a536-42e3-41fc-bf1c-ee35d171efdd"). InnerVolumeSpecName "kube-api-access-zddbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:03.921073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.921037    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t" pod "37d1bdca-7123-428e-9be1-61cd72e855a2" (UID: "37d1bdca-7123-428e-9be1-61cd72e855a2"). InnerVolumeSpecName "kube-api-access-d2j8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:03.994827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:03.994780    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:04.018226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.018191    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2j8t\" (UniqueName: \"kubernetes.io/projected/37d1bdca-7123-428e-9be1-61cd72e855a2-kube-api-access-d2j8t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:04.018226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.018225    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddbp\" (UniqueName: \"kubernetes.io/projected/b823a536-42e3-41fc-bf1c-ee35d171efdd-kube-api-access-zddbp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:04.110925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.110826    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:04.145004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.144968    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"]
Oct 30 07:21:04.149206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.149178    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-092c5c-r6vzr"]
Oct 30 07:21:04.242335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.241078    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:04.527567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.527533    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"
Oct 30 07:21:04.528088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.527541    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr" event={"ID":"b823a536-42e3-41fc-bf1c-ee35d171efdd","Type":"ContainerDied","Data":"59eabc0604f923b38e3bd7851735b4ec46868f054ff5aad57ed4d179ba2b4bb3"}
Oct 30 07:21:04.528088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.527724    6738 scope.go:122] "RemoveContainer" containerID="f3fe6d3d7006fa2401a6bc3b09f0446ec1d83be762a8828c4ccc621d42f981df"
Oct 30 07:21:04.529553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.529513    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" event={"ID":"105f610a-3b07-4433-807a-b851ad06e842","Type":"ContainerStarted","Data":"1f36aca6b883caf80f979683c99a5d879d4f7a89c8bb1e2445ba0ecb440c12fc"}
Oct 30 07:21:04.529728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.529574    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" event={"ID":"105f610a-3b07-4433-807a-b851ad06e842","Type":"ContainerStarted","Data":"749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e"}
Oct 30 07:21:04.529728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.529603    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:04.531694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.531661    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" event={"ID":"63074c69-da63-414a-8123-00065b48db97","Type":"ContainerStarted","Data":"7f1e73be2cf6b58b14b3d6f4d17a1f55ef4c7a917d9d67a307a815952d98d80d"}
Oct 30 07:21:04.531829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.531705    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" event={"ID":"63074c69-da63-414a-8123-00065b48db97","Type":"ContainerStarted","Data":"3ce24cd892695168fcf8f37f7afa0c3fc4ee970cb237842b9fe0a931ee02c255"}
Oct 30 07:21:04.531829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.531789    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:04.533507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.533483    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw" event={"ID":"37d1bdca-7123-428e-9be1-61cd72e855a2","Type":"ContainerDied","Data":"f2fe11e98a6bb8b43b94efffcab7fd0a5b56b5c924f1f55758bcc1146bb65c79"}
Oct 30 07:21:04.533652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.533495    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"
Oct 30 07:21:04.540610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.540593    6738 scope.go:122] "RemoveContainer" containerID="e8eb16adffbf8217de59f61afacba3048cda81c3324b2f5d84786f39bd9b384e"
Oct 30 07:21:04.601989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.601933    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" podStartSLOduration=2.601918623 podStartE2EDuration="2.601918623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:04.598481568 +0000 UTC m=+1425.031245461" watchObservedRunningTime="2026-10-30 07:21:04.601918623 +0000 UTC m=+1425.034682515"
Oct 30 07:21:04.617204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.617172    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:04.639159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.639097    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" podStartSLOduration=2.6390836220000002 podStartE2EDuration="2.639083622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:04.638513183 +0000 UTC m=+1425.071277075" watchObservedRunningTime="2026-10-30 07:21:04.639083622 +0000 UTC m=+1425.071847493"
Oct 30 07:21:04.677312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.677258    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:21:04.682977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.682947    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-092c5c-9c46fcc9b-s65dr"]
Oct 30 07:21:04.717073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.717038    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:21:04.721316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:04.721277    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-092c5c-76894d7bc9-zqmvw"]
Oct 30 07:21:05.692817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:05.692777    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d1bdca-7123-428e-9be1-61cd72e855a2" path="/var/lib/kubelet/pods/37d1bdca-7123-428e-9be1-61cd72e855a2/volumes"
Oct 30 07:21:05.693503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:05.693481    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9267eb-8e18-4113-9363-15ce62068d1c" path="/var/lib/kubelet/pods/3d9267eb-8e18-4113-9363-15ce62068d1c/volumes"
Oct 30 07:21:05.693962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:05.693942    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b823a536-42e3-41fc-bf1c-ee35d171efdd" path="/var/lib/kubelet/pods/b823a536-42e3-41fc-bf1c-ee35d171efdd/volumes"
Oct 30 07:21:06.077172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:06.077091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:07.521512 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:07.521470    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:21:08.206985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:08.206925    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:08.344988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:08.342202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:10.430647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:10.430612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:11.040822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:11.040778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:11.153408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:11.153374    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:11.401622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:11.401580    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:21:11.401845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:11.401816    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" containerID="cri-o://8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337" gracePeriod=30
Oct 30 07:21:11.457017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:11.456969    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:12.458370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:12.458331    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:13.019559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.019512    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:13.439683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.439649    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:21:13.440616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440592    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b823a536-42e3-41fc-bf1c-ee35d171efdd" containerName="payload-processing"
Oct 30 07:21:13.440616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440617    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b823a536-42e3-41fc-bf1c-ee35d171efdd" containerName="payload-processing"
Oct 30 07:21:13.440807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440675    6738 cpu_manager.go:414] "containerMap: removing container" podUID="37d1bdca-7123-428e-9be1-61cd72e855a2" containerName="payload-pre-processing"
Oct 30 07:21:13.440807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440686    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d1bdca-7123-428e-9be1-61cd72e855a2" containerName="payload-pre-processing"
Oct 30 07:21:13.440939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440887    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d1bdca-7123-428e-9be1-61cd72e855a2" containerName="payload-pre-processing"
Oct 30 07:21:13.440939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.440909    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b823a536-42e3-41fc-bf1c-ee35d171efdd" containerName="payload-processing"
Oct 30 07:21:13.444404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.444379    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.460008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.459978    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-45b148-openshift-default-dockercfg-j7nlw\""
Oct 30 07:21:13.479947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.479906    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:21:13.505042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.505006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:13.541375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541340    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbgq\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541456    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541509    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541544    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541609    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541803    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541851    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541910    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.541968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.541932    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.670320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.649026    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.670689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.670638    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.670900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.670878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.671198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.671120    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.671578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.671558    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.671732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.671716    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.671874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.671860    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.671985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.671972    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.672094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.672082    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbgq\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.693369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.672982    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.694224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.674664    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.694436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.675176    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.694547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.675406    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.694644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.676389    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.694733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.687452    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:13.694905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.687761    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.715096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.705747    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.800343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.798987    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbgq\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:13.813784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:13.813743    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\") pod \"e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") " pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:14.064724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.064620    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:14.401044 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:14.401007    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7979202_6d70_45ed_8ed5_8045828a4633.slice/crio-0d8b474c89a7437c7d191ee24c029d38e8f68d2a159f795a00cacf4d0246ad9c WatchSource:0}: Error finding container 0d8b474c89a7437c7d191ee24c029d38e8f68d2a159f795a00cacf4d0246ad9c: Status 404 returned error can't find the container with id 0d8b474c89a7437c7d191ee24c029d38e8f68d2a159f795a00cacf4d0246ad9c
Oct 30 07:21:14.401741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.401714    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:21:14.404091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.404046    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:14.404202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.404142    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:14.404202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.404191    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:14.412811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.412772    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:14.729080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.728997    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerStarted","Data":"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"}
Oct 30 07:21:14.729080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.729051    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerStarted","Data":"0d8b474c89a7437c7d191ee24c029d38e8f68d2a159f795a00cacf4d0246ad9c"}
Oct 30 07:21:14.776266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:14.776210    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podStartSLOduration=1.776191547 podStartE2EDuration="1.776191547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:14.774239421 +0000 UTC m=+1435.207003346" watchObservedRunningTime="2026-10-30 07:21:14.776191547 +0000 UTC m=+1435.208955439"
Oct 30 07:21:15.065841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.065756    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:15.071471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.071443    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:15.279892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.279847    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:21:15.280093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.279914    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:21:15.547772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.547737    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:21:15.547984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.547824    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:21:15.740837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.740800    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:15.742837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:15.742809    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:21:16.340816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.340773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:16.720837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.720802    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:21:16.750417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.750387    6738 generic.go:356] "Generic (PLEG): container finished" podID="d6a24618-7125-4e72-9da1-73900e657d6c" containerID="9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b" exitCode=0
Oct 30 07:21:16.751171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.750459    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerDied","Data":"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"}
Oct 30 07:21:16.751171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.750943    6738 scope.go:122] "RemoveContainer" containerID="9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"
Oct 30 07:21:16.752542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.752489    6738 generic.go:356] "Generic (PLEG): container finished" podID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerID="8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337" exitCode=0
Oct 30 07:21:16.752678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.752545    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerDied","Data":"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"}
Oct 30 07:21:16.752678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.752560    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"
Oct 30 07:21:16.752678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.752601    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf" event={"ID":"e27b7448-02ca-4d50-9961-1c5c4ef86f38","Type":"ContainerDied","Data":"1ababed8150ab72d3431436e4c4f2e9059f279a0b164f476a74c1d434088a31f"}
Oct 30 07:21:16.752678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.752634    6738 scope.go:122] "RemoveContainer" containerID="8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"
Oct 30 07:21:16.753904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.753873    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:16.753977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.753959    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:16.754015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.754001    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:16.802595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.802565    6738 scope.go:122] "RemoveContainer" containerID="45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"
Oct 30 07:21:16.807653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.807382    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:16.817163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.817140    6738 scope.go:122] "RemoveContainer" containerID="8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"
Oct 30 07:21:16.817611 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:16.817588    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337\": container with ID starting with 8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337 not found: ID does not exist" containerID="8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"
Oct 30 07:21:16.817738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.817624    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337"} err="failed to get container status \"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337\": rpc error: code = NotFound desc = could not find container \"8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337\": container with ID starting with 8757cebfe94f6f36e33f75c2bfe31fff857e95576feb58a74fdabd09d6732337 not found: ID does not exist"
Oct 30 07:21:16.817738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.817648    6738 scope.go:122] "RemoveContainer" containerID="45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"
Oct 30 07:21:16.817962 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:16.817939    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066\": container with ID starting with 45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066 not found: ID does not exist" containerID="45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"
Oct 30 07:21:16.818005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.817974    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066"} err="failed to get container status \"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066\": rpc error: code = NotFound desc = could not find container \"45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066\": container with ID starting with 45eeb3c0639f8a26587f2ef01d3ba69be8302ef52de2dcbe90feba161a1a6066 not found: ID does not exist"
Oct 30 07:21:16.854917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.854852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.854949    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.854983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855017    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855058    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855143    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855184    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.855499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.855456    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\") pod \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\" (UID: \"e27b7448-02ca-4d50-9961-1c5c4ef86f38\") "
Oct 30 07:21:16.856825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.856793    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:21:16.858114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.858080    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:21:16.859106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.859083    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:16.859583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.859320    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:21:16.859774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.859744    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:21:16.862981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.862940    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:21:16.863935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.863901    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:21:16.866944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.866120    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "kube-api-access-n67sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:16.866944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.866895    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token" pod "e27b7448-02ca-4d50-9961-1c5c4ef86f38" (UID: "e27b7448-02ca-4d50-9961-1c5c4ef86f38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:16.957392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957357    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957392    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957409    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67sg\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-kube-api-access-n67sg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957426    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957438    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957449    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957460    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7448-02ca-4d50-9961-1c5c4ef86f38-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957474    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:16.957603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:16.957488    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7448-02ca-4d50-9961-1c5c4ef86f38-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:17.103315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.103259    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:21:17.114210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.114136    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-092c5c-openshift-default-7fb5b4b57-kwwdf"]
Oct 30 07:21:17.345245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.345199    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:17.506716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.506672    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:17.700777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.700697    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" path="/var/lib/kubelet/pods/e27b7448-02ca-4d50-9961-1c5c4ef86f38/volumes"
Oct 30 07:21:17.762500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.762468    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerStarted","Data":"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"}
Oct 30 07:21:17.987530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:17.987408    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:21:18.490318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:18.490042    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:21:18.493343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:18.493281    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 07:21:18.493493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:18.493368    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 07:21:18.555688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:18.555648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:19.321550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.321515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:19.435358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.435309    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:19.493417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.493378    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 07:21:19.493656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.493459    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 07:21:19.800670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.799933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:19.921443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.921408    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:19.950813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.950772    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:21:19.951514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951492    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.951514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951514    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.951697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951529    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.951697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951536    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.951803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951695    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.951803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.951711    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27b7448-02ca-4d50-9961-1c5c4ef86f38" containerName="istio-proxy"
Oct 30 07:21:19.955417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:19.955391    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.001786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.001750    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:21:20.015352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.012957    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-45b148-dockercfg-xhrdf\""
Oct 30 07:21:20.015352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.013536    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-45b148\""
Oct 30 07:21:20.117453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.115146    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.117453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.115215    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9qg\" (UniqueName: \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.220601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.189684    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:21:20.220601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.197529    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.237435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.236462    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbjc\" (UniqueName: \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.237435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.236548    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.237435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.236610    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9qg\" (UniqueName: \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.237435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.236801    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.238619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.238593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.253555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.253528    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:21:20.280557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.264145    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9qg\" (UniqueName: \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\") pod \"payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.291745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.290543    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:20.338387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.338347    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbjc\" (UniqueName: \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.338851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.338504    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.339356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.339331    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.363948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.363909    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbjc\" (UniqueName: \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\") pod \"payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.513316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.496754    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 07:21:20.513316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.496823    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 07:21:20.579105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.578637    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:20.676658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.676622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:20.681744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.681712    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:21:20.699563 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:20.699534    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7313b357_1114_443d_a10e_de4adc88ba35.slice/crio-c9f31b11c04e0a6dbf1a1b38fecaa318b2bd80599ee8e89b7013740b740d1610 WatchSource:0}: Error finding container c9f31b11c04e0a6dbf1a1b38fecaa318b2bd80599ee8e89b7013740b740d1610: Status 404 returned error can't find the container with id c9f31b11c04e0a6dbf1a1b38fecaa318b2bd80599ee8e89b7013740b740d1610
Oct 30 07:21:20.794718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.793761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:20.822141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.822037    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" event={"ID":"7313b357-1114-443d-a10e-de4adc88ba35","Type":"ContainerStarted","Data":"c9f31b11c04e0a6dbf1a1b38fecaa318b2bd80599ee8e89b7013740b740d1610"}
Oct 30 07:21:20.995632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.995490    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:20.999490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:20.999457    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:21:21.492733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.492689    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 07:21:21.493153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.492756    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 07:21:21.838390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.838269    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" event={"ID":"ce43a838-0e27-4eb0-b183-198cd490250f","Type":"ContainerStarted","Data":"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"}
Oct 30 07:21:21.838390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.838378    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" event={"ID":"ce43a838-0e27-4eb0-b183-198cd490250f","Type":"ContainerStarted","Data":"76eade9daa4febe177526df96a1356d79a69c262c6ccfc32ff95048ea4e448c2"}
Oct 30 07:21:21.838822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.838753    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:21.840273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.840249    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" event={"ID":"7313b357-1114-443d-a10e-de4adc88ba35","Type":"ContainerStarted","Data":"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"}
Oct 30 07:21:21.840430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.840324    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:21.893323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.893232    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" podStartSLOduration=1.893211524 podStartE2EDuration="1.893211524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:21.88656727 +0000 UTC m=+1442.319331164" watchObservedRunningTime="2026-10-30 07:21:21.893211524 +0000 UTC m=+1442.325975420"
Oct 30 07:21:21.928829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:21.928763    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" podStartSLOduration=2.92875243 podStartE2EDuration="2.92875243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:21.928162777 +0000 UTC m=+1442.360926671" watchObservedRunningTime="2026-10-30 07:21:21.92875243 +0000 UTC m=+1442.361516323"
Oct 30 07:21:22.490726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:22.490688    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 07:21:22.491049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:22.491017    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 07:21:23.024984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:23.024944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:23.493610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:23.493579    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:21:23.494477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:23.494457    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:21:24.227853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:24.227821    6738 scope.go:122] "RemoveContainer" containerID="ee813825d5dd0df799c8349e8c0e64f0b3ec90a2ebcf48ad782205b611ba26a4"
Oct 30 07:21:24.238573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:24.238548    6738 scope.go:122] "RemoveContainer" containerID="792b9e02bc587c950e59aa138f8ce2c816c08aef9dd1fc19ac6cf53e956e45d4"
Oct 30 07:21:24.807264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:24.807226    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:24.956453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:24.956420    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:25.089577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.089482    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:25.429483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.429448    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:21:25.462071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.462030    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:21:25.579468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.579432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:25.721488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.712933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:25.841149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:25.838090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:26.735865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:26.735810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:28.529484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:28.529434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:30.060207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.060077    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:30.429849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.429768    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" podUID="288bcd04-57c1-4766-a648-f9673e990f02" containerName="payload-processing" containerID="cri-o://0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606" gracePeriod=25
Oct 30 07:21:30.455814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.455776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:30.462683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.462631    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" podUID="74982c3c-b845-44f9-937c-e2e7566b7a73" containerName="payload-pre-processing" containerID="cri-o://c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91" gracePeriod=25
Oct 30 07:21:30.735974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.735945    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:21:30.764839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.764813    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:21:30.865338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.865283    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\") pod \"288bcd04-57c1-4766-a648-f9673e990f02\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") "
Oct 30 07:21:30.865541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.865462    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\" (UniqueName: \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\") pod \"288bcd04-57c1-4766-a648-f9673e990f02\" (UID: \"288bcd04-57c1-4766-a648-f9673e990f02\") "
Oct 30 07:21:30.865541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.865528    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\" (UniqueName: \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\") pod \"74982c3c-b845-44f9-937c-e2e7566b7a73\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") "
Oct 30 07:21:30.865664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.865557    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\") pod \"74982c3c-b845-44f9-937c-e2e7566b7a73\" (UID: \"74982c3c-b845-44f9-937c-e2e7566b7a73\") "
Oct 30 07:21:30.865749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.865718    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume" pod "288bcd04-57c1-4766-a648-f9673e990f02" (UID: "288bcd04-57c1-4766-a648-f9673e990f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:30.866103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.866075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume" pod "74982c3c-b845-44f9-937c-e2e7566b7a73" (UID: "74982c3c-b845-44f9-937c-e2e7566b7a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:30.866280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.866258    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288bcd04-57c1-4766-a648-f9673e990f02-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:30.866386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.866284    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74982c3c-b845-44f9-937c-e2e7566b7a73-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:30.867930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.867910    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq" pod "74982c3c-b845-44f9-937c-e2e7566b7a73" (UID: "74982c3c-b845-44f9-937c-e2e7566b7a73"). InnerVolumeSpecName "kube-api-access-d6qwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:30.868018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.867938    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc" pod "288bcd04-57c1-4766-a648-f9673e990f02" (UID: "288bcd04-57c1-4766-a648-f9673e990f02"). InnerVolumeSpecName "kube-api-access-cs4rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:30.967392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967287    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4rc\" (UniqueName: \"kubernetes.io/projected/288bcd04-57c1-4766-a648-f9673e990f02-kube-api-access-cs4rc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:30.967392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967336    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qwq\" (UniqueName: \"kubernetes.io/projected/74982c3c-b845-44f9-937c-e2e7566b7a73-kube-api-access-d6qwq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:30.967692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967427    6738 generic.go:356] "Generic (PLEG): container finished" podID="74982c3c-b845-44f9-937c-e2e7566b7a73" containerID="c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91" exitCode=0
Oct 30 07:21:30.967692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967490    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"
Oct 30 07:21:30.967692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967509    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" event={"ID":"74982c3c-b845-44f9-937c-e2e7566b7a73","Type":"ContainerDied","Data":"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"}
Oct 30 07:21:30.967692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967568    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd" event={"ID":"74982c3c-b845-44f9-937c-e2e7566b7a73","Type":"ContainerDied","Data":"e33f04a2514d42791cd8eda2678917fbf694824b0769e1d80911e891f5de9ff6"}
Oct 30 07:21:30.967692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.967592    6738 scope.go:122] "RemoveContainer" containerID="c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"
Oct 30 07:21:30.969155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.969131    6738 generic.go:356] "Generic (PLEG): container finished" podID="288bcd04-57c1-4766-a648-f9673e990f02" containerID="0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606" exitCode=0
Oct 30 07:21:30.969332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.969163    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" event={"ID":"288bcd04-57c1-4766-a648-f9673e990f02","Type":"ContainerDied","Data":"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"}
Oct 30 07:21:30.969332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.969184    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"
Oct 30 07:21:30.969332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.969196    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf" event={"ID":"288bcd04-57c1-4766-a648-f9673e990f02","Type":"ContainerDied","Data":"f1a709586906dfa7090ffd3e8c617d3a5cbba6126bba98d4deab85ef8ca83e04"}
Oct 30 07:21:30.978421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.978398    6738 scope.go:122] "RemoveContainer" containerID="c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"
Oct 30 07:21:30.978748 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:30.978729    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91\": container with ID starting with c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91 not found: ID does not exist" containerID="c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"
Oct 30 07:21:30.978832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.978755    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91"} err="failed to get container status \"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91\": rpc error: code = NotFound desc = could not find container \"c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91\": container with ID starting with c2bf71d018adeed5c2058be1b7b49bea7fad26f1bc70d21db354f62f6a0a8b91 not found: ID does not exist"
Oct 30 07:21:30.978832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:30.978772    6738 scope.go:122] "RemoveContainer" containerID="0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"
Oct 30 07:21:31.011893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.011866    6738 scope.go:122] "RemoveContainer" containerID="0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"
Oct 30 07:21:31.012220 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:31.012198    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606\": container with ID starting with 0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606 not found: ID does not exist" containerID="0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"
Oct 30 07:21:31.012275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.012231    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606"} err="failed to get container status \"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606\": rpc error: code = NotFound desc = could not find container \"0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606\": container with ID starting with 0610e2fdd0014929133deea3bd4fd78f31e7059c27b236f9ec9d239702038606 not found: ID does not exist"
Oct 30 07:21:31.040726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.040689    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:21:31.047357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.047328    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-d4d5667dc-4r4zd"]
Oct 30 07:21:31.080816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.080776    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:21:31.084907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.084872    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-5d67bcf95b-7mxvf"]
Oct 30 07:21:31.700463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.700385    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288bcd04-57c1-4766-a648-f9673e990f02" path="/var/lib/kubelet/pods/288bcd04-57c1-4766-a648-f9673e990f02/volumes"
Oct 30 07:21:31.700865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:31.700842    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74982c3c-b845-44f9-937c-e2e7566b7a73" path="/var/lib/kubelet/pods/74982c3c-b845-44f9-937c-e2e7566b7a73/volumes"
Oct 30 07:21:32.425050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:32.425015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:32.852144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:32.852054    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:21:32.852349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:32.852328    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:21:34.589337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:34.589281    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:36.721240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:36.721202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:37.519577 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:21:37.519542    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:21:38.574507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:38.574468    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:38.768837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:38.768801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:40.014703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.014650    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:40.550204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.550167    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:21:40.550939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.550903    6738 cpu_manager.go:414] "containerMap: removing container" podUID="74982c3c-b845-44f9-937c-e2e7566b7a73" containerName="payload-pre-processing"
Oct 30 07:21:40.550939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.550928    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74982c3c-b845-44f9-937c-e2e7566b7a73" containerName="payload-pre-processing"
Oct 30 07:21:40.551132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.550983    6738 cpu_manager.go:414] "containerMap: removing container" podUID="288bcd04-57c1-4766-a648-f9673e990f02" containerName="payload-processing"
Oct 30 07:21:40.551132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.550993    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288bcd04-57c1-4766-a648-f9673e990f02" containerName="payload-processing"
Oct 30 07:21:40.551132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.551131    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="288bcd04-57c1-4766-a648-f9673e990f02" containerName="payload-processing"
Oct 30 07:21:40.551259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.551147    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="74982c3c-b845-44f9-937c-e2e7566b7a73" containerName="payload-pre-processing"
Oct 30 07:21:40.553659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.553642    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.581969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.581605    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:40.583530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.583500    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:21:40.591206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.591177    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-411c9d-openshift-default-dockercfg-jdhn8\""
Oct 30 07:21:40.629970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.629933    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.629976    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630011    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630146    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630415    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630485    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630518    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwjw\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630547    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.630637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.630614    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.783928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.783886    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.784203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.784180    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.784617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.784599    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.784794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.784777    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.784921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.784903    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwjw\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.785030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.785017    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.785136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.785123    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.785274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.785262    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.785427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.785408    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.812317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.785998    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.812598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.800708    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.812699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.801414    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.813892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.813860    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.841654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.827616    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:40.841654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.828821    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.846414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.846375    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.847286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.847257    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.859634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.851853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.900384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.894192    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwjw\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\") pod \"e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") " pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:40.945537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:40.945499    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:41.073178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.073145    6738 generic.go:356] "Generic (PLEG): container finished" podID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerID="9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5" exitCode=0
Oct 30 07:21:41.073661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.073227    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerDied","Data":"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"}
Oct 30 07:21:41.073744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.073729    6738 scope.go:122] "RemoveContainer" containerID="9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"
Oct 30 07:21:41.075394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.075358    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:41.075507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.075432    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:41.075507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.075464    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:41.174480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.174446    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:41.533585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.530692    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:21:41.539245 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:41.539209    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a45d26_7fac_475b_91a4_c305f67aa821.slice/crio-dc0b204ee2b841dd90beda1f19ef2bcff3b550d30831221b926926ce3432a036 WatchSource:0}: Error finding container dc0b204ee2b841dd90beda1f19ef2bcff3b550d30831221b926926ce3432a036: Status 404 returned error can't find the container with id dc0b204ee2b841dd90beda1f19ef2bcff3b550d30831221b926926ce3432a036
Oct 30 07:21:41.542667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.542623    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:41.542790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.542719    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:41.542790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:41.542765    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:42.098370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.098336    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerStarted","Data":"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"}
Oct 30 07:21:42.100358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.100328    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerStarted","Data":"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"}
Oct 30 07:21:42.100495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.100380    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerStarted","Data":"dc0b204ee2b841dd90beda1f19ef2bcff3b550d30831221b926926ce3432a036"}
Oct 30 07:21:42.191333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.188582    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:42.199163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.199129    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:42.209867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.209815    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podStartSLOduration=2.209802038 podStartE2EDuration="2.209802038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:42.206884973 +0000 UTC m=+1462.639648866" watchObservedRunningTime="2026-10-30 07:21:42.209802038 +0000 UTC m=+1462.642565931"
Oct 30 07:21:42.342321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.339130    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:21:42.367388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.353442    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:42.367388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.353508    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:42.713501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:42.713463    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:43.147316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:43.146708    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:43.149439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:43.149414    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:21:43.338099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:43.338060    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:43.338372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:43.338135    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:44.338210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:44.338169    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:44.338708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:44.338245    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:44.701178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:44.701138    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:44.833448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:44.833414    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:44.961075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:44.960978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:45.337984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:45.337888    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:45.337984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:45.337953    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:46.336420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.336385    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:21:46.337531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.337507    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:46.337595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.337569    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:46.612259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.612010    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:46.734473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.734436    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:46.754730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.754699    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:21:46.781681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.781643    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.814848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.814801    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-411c9d-dockercfg-g69rw\""
Oct 30 07:21:46.815093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.815029    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-411c9d\""
Oct 30 07:21:46.818770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.818745    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:21:46.824493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.824463    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:21:46.829239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.829212    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:46.841262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.841228    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:21:46.842364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.842332    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.842499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.842399    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wncwr\" (UniqueName: \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.866385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.865156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:46.943701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.943665    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:46.943885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.943756    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d48\" (UniqueName: \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:46.943938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.943868    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.943938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.943916    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wncwr\" (UniqueName: \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.944590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.944571    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:46.976320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:46.967395    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wncwr\" (UniqueName: \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\") pod \"payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:47.047263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.047216    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:47.047679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.047648    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d48\" (UniqueName: \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:47.049024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.048995    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:47.096112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.096077    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d48\" (UniqueName: \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\") pod \"payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:47.117943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.117856    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:47.161602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.161563    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:47.242119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.238890    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:21:47.252034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.250624    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.326213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.325336    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8d44bf-dockercfg-blhnq\""
Oct 30 07:21:47.326213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.325499    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8d44bf\""
Oct 30 07:21:47.328792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.328759    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:21:47.337708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.337674    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 07:21:47.338191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.337724    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 07:21:47.402380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.366790    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbc7\" (UniqueName: \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.402380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.366897    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.454351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.451411    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:21:47.456132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.456098    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:21:47.456282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.456202    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.557583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.548909    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mn7\" (UniqueName: \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.557583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.549055    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.557583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.549224    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbc7\" (UniqueName: \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.557583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.549321    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.557583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.550193    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.662712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.654279    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mn7\" (UniqueName: \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.662712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.654433    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.662712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.655720    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.669867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.666191    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbc7\" (UniqueName: \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\") pod \"payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.669867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.666614    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:47.834880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.834809    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mn7\" (UniqueName: \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\") pod \"payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.842222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.841963    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:47.967317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:47.964908    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:21:48.016610 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:48.016576    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fceb53_e30c_4233_b8d7_33996640a59d.slice/crio-65507570715edc5266cb3bad55b87bf1682bbd7cae76afc883f9358c9def242c WatchSource:0}: Error finding container 65507570715edc5266cb3bad55b87bf1682bbd7cae76afc883f9358c9def242c: Status 404 returned error can't find the container with id 65507570715edc5266cb3bad55b87bf1682bbd7cae76afc883f9358c9def242c
Oct 30 07:21:48.030311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.030012    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:21:48.221596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.221504    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:21:48.225736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.225057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" event={"ID":"c8fceb53-e30c-4233-b8d7-33996640a59d","Type":"ContainerStarted","Data":"65507570715edc5266cb3bad55b87bf1682bbd7cae76afc883f9358c9def242c"}
Oct 30 07:21:48.225736 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:48.225165    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf2d105_82e6_4464_a75f_9d484e89b3b6.slice/crio-ae632622cd2e5dcd6335558a7afda724d9c5b367de17eeac3c388cea3484e5ae WatchSource:0}: Error finding container ae632622cd2e5dcd6335558a7afda724d9c5b367de17eeac3c388cea3484e5ae: Status 404 returned error can't find the container with id ae632622cd2e5dcd6335558a7afda724d9c5b367de17eeac3c388cea3484e5ae
Oct 30 07:21:48.227450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.227405    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" event={"ID":"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652","Type":"ContainerStarted","Data":"0d98da672ed060504b5563b2f418c1351ed40cd7b7960ccccdbe6a355572cebf"}
Oct 30 07:21:48.228280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.228139    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:21:48.247889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.246583    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:48.281316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.274833    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:21:48.281316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.278226    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" podStartSLOduration=2.278204837 podStartE2EDuration="2.278204837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:48.275257338 +0000 UTC m=+1468.708021232" watchObservedRunningTime="2026-10-30 07:21:48.278204837 +0000 UTC m=+1468.710968732"
Oct 30 07:21:48.349589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.347880    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:21:48.352906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:48.352883    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:21:49.238213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.238164    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" event={"ID":"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652","Type":"ContainerStarted","Data":"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"}
Oct 30 07:21:49.241190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.240286    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" event={"ID":"b71bc68b-2bcd-4118-a61f-25516b5e31cc","Type":"ContainerStarted","Data":"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"}
Oct 30 07:21:49.241190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.240571    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:21:49.241190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.240613    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" event={"ID":"b71bc68b-2bcd-4118-a61f-25516b5e31cc","Type":"ContainerStarted","Data":"32c351baa3159dd9cd0132a0aa78be8d55f376b919884244bb506938dcdf5771"}
Oct 30 07:21:49.242690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.242662    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" event={"ID":"c8fceb53-e30c-4233-b8d7-33996640a59d","Type":"ContainerStarted","Data":"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"}
Oct 30 07:21:49.242847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.242753    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:21:49.244512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.244490    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" event={"ID":"7bf2d105-82e6-4464-a75f-9d484e89b3b6","Type":"ContainerStarted","Data":"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"}
Oct 30 07:21:49.244622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.244531    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" event={"ID":"7bf2d105-82e6-4464-a75f-9d484e89b3b6","Type":"ContainerStarted","Data":"ae632622cd2e5dcd6335558a7afda724d9c5b367de17eeac3c388cea3484e5ae"}
Oct 30 07:21:49.244622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.244551    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:21:49.302821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.302759    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" podStartSLOduration=2.302741669 podStartE2EDuration="2.302741669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:49.285709618 +0000 UTC m=+1469.718473534" watchObservedRunningTime="2026-10-30 07:21:49.302741669 +0000 UTC m=+1469.735505599"
Oct 30 07:21:49.328225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.328160    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" podStartSLOduration=2.32814004 podStartE2EDuration="2.32814004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:49.326898476 +0000 UTC m=+1469.759662370" watchObservedRunningTime="2026-10-30 07:21:49.32814004 +0000 UTC m=+1469.760903930"
Oct 30 07:21:49.380641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:49.380576    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" podStartSLOduration=3.38055269 podStartE2EDuration="3.38055269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:49.36732925 +0000 UTC m=+1469.800093136" watchObservedRunningTime="2026-10-30 07:21:49.38055269 +0000 UTC m=+1469.813316583"
Oct 30 07:21:50.235337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.235286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:50.732441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.732364    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"]
Oct 30 07:21:50.735915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.735891    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.753699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.753661    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"]
Oct 30 07:21:50.761554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.761509    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:21:50.761554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.761529    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:21:50.761787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.761673    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:21:50.762967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.762945    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:21:50.841739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.841707    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.841935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.841912    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlhq\" (UniqueName: \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.841986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.841962    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.943064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.943028    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlhq\" (UniqueName: \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.943273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.943079    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.943273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.943159    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.943884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.943849    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.945617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.945596    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:50.966847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:50.966803    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlhq\" (UniqueName: \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\") pod \"maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:51.051349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:51.051250    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:51.405797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:51.405760    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"]
Oct 30 07:21:52.298379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:52.298035    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr" event={"ID":"4299cf55-9fa2-4530-a6bd-5b99026b7b25","Type":"ContainerStarted","Data":"a1626e082256d66e97e5312618af2a0df510d049cefdcf868c778c68a6de50b8"}
Oct 30 07:21:52.298379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:52.298114    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr" event={"ID":"4299cf55-9fa2-4530-a6bd-5b99026b7b25","Type":"ContainerStarted","Data":"e1bbc9a76270fa7d42e82508e0c40525db5f4f4d7960b228732a4cb91b9cbc83"}
Oct 30 07:21:52.333704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:52.333668    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:52.341170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:52.341056    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr" podStartSLOduration=2.341039713 podStartE2EDuration="2.341039713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:52.340265415 +0000 UTC m=+1472.773029305" watchObservedRunningTime="2026-10-30 07:21:52.341039713 +0000 UTC m=+1472.773803608"
Oct 30 07:21:53.307698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:53.307599    6738 generic.go:356] "Generic (PLEG): container finished" podID="4299cf55-9fa2-4530-a6bd-5b99026b7b25" containerID="a1626e082256d66e97e5312618af2a0df510d049cefdcf868c778c68a6de50b8" exitCode=0
Oct 30 07:21:53.307698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:53.307680    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr" event={"ID":"4299cf55-9fa2-4530-a6bd-5b99026b7b25","Type":"ContainerDied","Data":"a1626e082256d66e97e5312618af2a0df510d049cefdcf868c778c68a6de50b8"}
Oct 30 07:21:54.355550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.355513    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:54.557619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.557587    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:54.658513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.658479    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\") pod \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") "
Oct 30 07:21:54.658796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.658740    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\" (UniqueName: \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\") pod \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") "
Oct 30 07:21:54.658904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.658798    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\") pod \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\" (UID: \"4299cf55-9fa2-4530-a6bd-5b99026b7b25\") "
Oct 30 07:21:54.659597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.659566    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca" pod "4299cf55-9fa2-4530-a6bd-5b99026b7b25" (UID: "4299cf55-9fa2-4530-a6bd-5b99026b7b25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:21:54.660868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.660836    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca" pod "4299cf55-9fa2-4530-a6bd-5b99026b7b25" (UID: "4299cf55-9fa2-4530-a6bd-5b99026b7b25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:21:54.661397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.661370    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq" pod "4299cf55-9fa2-4530-a6bd-5b99026b7b25" (UID: "4299cf55-9fa2-4530-a6bd-5b99026b7b25"). InnerVolumeSpecName "kube-api-access-tqlhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:21:54.761077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.761043    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlhq\" (UniqueName: \"kubernetes.io/projected/4299cf55-9fa2-4530-a6bd-5b99026b7b25-kube-api-access-tqlhq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:54.761077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.761076    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:54.761278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:54.761093    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4299cf55-9fa2-4530-a6bd-5b99026b7b25-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:21:55.333639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:55.333605    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"
Oct 30 07:21:55.333856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:55.333814    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr" event={"ID":"4299cf55-9fa2-4530-a6bd-5b99026b7b25","Type":"ContainerDied","Data":"e1bbc9a76270fa7d42e82508e0c40525db5f4f4d7960b228732a4cb91b9cbc83"}
Oct 30 07:21:55.333938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:55.333922    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1bbc9a76270fa7d42e82508e0c40525db5f4f4d7960b228732a4cb91b9cbc83"
Oct 30 07:21:56.399951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.399915    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:56.852783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.852674    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:21:56.853462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.853438    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4299cf55-9fa2-4530-a6bd-5b99026b7b25" containerName="revoke-keys"
Oct 30 07:21:56.853462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.853454    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4299cf55-9fa2-4530-a6bd-5b99026b7b25" containerName="revoke-keys"
Oct 30 07:21:56.853592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.853565    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4299cf55-9fa2-4530-a6bd-5b99026b7b25" containerName="revoke-keys"
Oct 30 07:21:56.856167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.856150    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.889542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.889510    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:21:56.990928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.990889    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.990948    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rpv\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991162    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991334    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991391    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991479    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991527    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:56.991615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:56.991576    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.031746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.031712    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.095836    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.095902    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.095948    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.095979    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.096029    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.096095    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.096154    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.096276    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rpv\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.096382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.097579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.097116    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.098310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.097765    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.098310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.098001    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.098310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.098234    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.101124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.100782    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.124471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.124435    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.125514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.125479    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.127511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.127452    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.127683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.127662    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rpv\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.170095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.170057    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:57.350495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.350463    6738 generic.go:356] "Generic (PLEG): container finished" podID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerID="577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf" exitCode=0
Oct 30 07:21:57.350684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.350544    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerDied","Data":"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"}
Oct 30 07:21:57.351070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.351052    6738 scope.go:122] "RemoveContainer" containerID="577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"
Oct 30 07:21:57.352819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.352781    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:57.352959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.352879    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:57.352959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.352931    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:57.495979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.495949    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:21:57.498423 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:21:57.498389    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d51648_68a0_4881_b123_3dde837126c1.slice/crio-02745c1fd135a3de50a735e567a86c1a8e0d96f25a2bdde3a88a0cd4444ad1de WatchSource:0}: Error finding container 02745c1fd135a3de50a735e567a86c1a8e0d96f25a2bdde3a88a0cd4444ad1de: Status 404 returned error can't find the container with id 02745c1fd135a3de50a735e567a86c1a8e0d96f25a2bdde3a88a0cd4444ad1de
Oct 30 07:21:57.501634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.501592    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:57.501790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.501697    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:57.501790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:57.501750    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:21:58.298832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.298796    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:21:58.394758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.394713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" event={"ID":"46d51648-68a0-4881-b123-3dde837126c1","Type":"ContainerStarted","Data":"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"}
Oct 30 07:21:58.396370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.396163    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" event={"ID":"46d51648-68a0-4881-b123-3dde837126c1","Type":"ContainerStarted","Data":"02745c1fd135a3de50a735e567a86c1a8e0d96f25a2bdde3a88a0cd4444ad1de"}
Oct 30 07:21:58.401658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.401625    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerStarted","Data":"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"}
Oct 30 07:21:58.445872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.445818    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podStartSLOduration=2.445804725 podStartE2EDuration="2.445804725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:58.443118369 +0000 UTC m=+1478.875882263" watchObservedRunningTime="2026-10-30 07:21:58.445804725 +0000 UTC m=+1478.878568618"
Oct 30 07:21:58.936992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.936958    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:21:58.938120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.938081    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:21:58.938258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:58.938143    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:21:59.193377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.192325    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:21:59.205539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.205485    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:21:59.205847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.205799    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:21:59.330429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.330388    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"]
Oct 30 07:21:59.335101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.335072    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.351117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.351065    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"]
Oct 30 07:21:59.359735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.359518    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:21:59.359735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.359608    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:21:59.359735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.359687    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:21:59.359735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.359690    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:21:59.430637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.430596    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ggp\" (UniqueName: \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.430812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.430657    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.430857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.430822    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.491382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.481400    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:21:59.532078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.532029    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.532266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.532218    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ggp\" (UniqueName: \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.532415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.532335    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.533115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.533082    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.537328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.535049    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.556335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.556280    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ggp\" (UniqueName: \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\") pod \"maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.654028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.653991    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:21:59.938598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.938565    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:21:59.939045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:21:59.938629    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:22:00.045153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.045118    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"]
Oct 30 07:22:00.049321 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:00.048139    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b04ea9_2a60_46fc_b988_d8f0f2912578.slice/crio-9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329 WatchSource:0}: Error finding container 9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329: Status 404 returned error can't find the container with id 9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329
Oct 30 07:22:00.194085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.194049    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:22:00.194407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.194374    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:22:00.256223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.256198    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:22:00.257356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.257332    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:22:00.257508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.257489    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:22:00.258866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.258841    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:22:00.446371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.446331    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj" event={"ID":"35b04ea9-2a60-46fc-b988-d8f0f2912578","Type":"ContainerStarted","Data":"9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329"}
Oct 30 07:22:00.529919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.529861    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:00.936712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.936631    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:22:00.936960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:00.936936    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:22:01.175522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.175483    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:22:01.176148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.176112    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:22:01.239414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.239050    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:22:01.357759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.357514    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:22:01.429715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.429500    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:01.462995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.462951    6738 generic.go:356] "Generic (PLEG): container finished" podID="35b04ea9-2a60-46fc-b988-d8f0f2912578" containerID="1adccc871629592f2efd4b7ec492445ed8b0a56ba4c505c01728ef47f38c781e" exitCode=0
Oct 30 07:22:01.463210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.463018    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj" event={"ID":"35b04ea9-2a60-46fc-b988-d8f0f2912578","Type":"ContainerDied","Data":"1adccc871629592f2efd4b7ec492445ed8b0a56ba4c505c01728ef47f38c781e"}
Oct 30 07:22:01.606374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.605391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:01.938148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.937727    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:22:01.938148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:01.937789    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:22:02.173320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.171762    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:22:02.173320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.171842    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:22:02.816237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.816140    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:02.816997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.816970    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:22:02.945317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.934900    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\") pod \"35b04ea9-2a60-46fc-b988-d8f0f2912578\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") "
Oct 30 07:22:02.945317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.934977    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\") pod \"35b04ea9-2a60-46fc-b988-d8f0f2912578\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") "
Oct 30 07:22:02.945317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.935194    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\" (UniqueName: \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\") pod \"35b04ea9-2a60-46fc-b988-d8f0f2912578\" (UID: \"35b04ea9-2a60-46fc-b988-d8f0f2912578\") "
Oct 30 07:22:02.945317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.944792    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca" pod "35b04ea9-2a60-46fc-b988-d8f0f2912578" (UID: "35b04ea9-2a60-46fc-b988-d8f0f2912578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:02.947321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:02.945968    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:03.003983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.003929    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca" pod "35b04ea9-2a60-46fc-b988-d8f0f2912578" (UID: "35b04ea9-2a60-46fc-b988-d8f0f2912578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:22:03.012034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.010804    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:22:03.012034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.010865    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:22:03.013584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.013521    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:03.050540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.050495    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp" pod "35b04ea9-2a60-46fc-b988-d8f0f2912578" (UID: "35b04ea9-2a60-46fc-b988-d8f0f2912578"). InnerVolumeSpecName "kube-api-access-l9ggp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:03.050789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.050771    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b04ea9-2a60-46fc-b988-d8f0f2912578-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:03.152505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.152468    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ggp\" (UniqueName: \"kubernetes.io/projected/35b04ea9-2a60-46fc-b988-d8f0f2912578-kube-api-access-l9ggp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:03.175321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.171714    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:22:03.175321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.171781    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:22:03.235420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.231801    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:22:03.235420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.232497    6738 cpu_manager.go:414] "containerMap: removing container" podUID="35b04ea9-2a60-46fc-b988-d8f0f2912578" containerName="revoke-keys"
Oct 30 07:22:03.235420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.232511    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b04ea9-2a60-46fc-b988-d8f0f2912578" containerName="revoke-keys"
Oct 30 07:22:03.235420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.232626    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b04ea9-2a60-46fc-b988-d8f0f2912578" containerName="revoke-keys"
Oct 30 07:22:03.237090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.237066    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.255645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.255577    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-89664e-openshift-default-dockercfg-dqtgv\""
Oct 30 07:22:03.264705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.263624    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:22:03.325391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.313916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:03.378399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378123    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378208    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378288    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378335    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378488    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378519    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378606    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378686    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.378837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.378722    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfkq\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.565365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.564993    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.565365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.565070    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.579316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.569503    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.579316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.569566    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.579316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.576271    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.579316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.577542    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.606319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.577588    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.619774    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.621499    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.621629    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.621680    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfkq\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.621766    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.622312    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.622750    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.624379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.623079    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.632057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.631919    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.634377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.633684    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.644383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.643594    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj" event={"ID":"35b04ea9-2a60-46fc-b988-d8f0f2912578","Type":"ContainerDied","Data":"9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329"}
Oct 30 07:22:03.644383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.643658    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed8325fcd3e8272e8be54d5b847d4098f985d6634b4bc626ea00a72e5db0329"
Oct 30 07:22:03.644383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.643732    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"
Oct 30 07:22:03.667829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.667755    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfkq\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\") pod \"e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") " pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.751381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.751136    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:03.860061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.859726    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:03.937403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.937371    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 07:22:03.937651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:03.937614    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 07:22:04.204928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.204444    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 07:22:04.204928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.204514    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 07:22:04.334537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.334485    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:22:04.337253 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:04.337147    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc1d1cb_09d8_4d01_94e6_6af079b5c736.slice/crio-b36ed3d5bdfb09e96f82f8e84297d1acce8c32f3e6559849c6cc9bb80da1de82 WatchSource:0}: Error finding container b36ed3d5bdfb09e96f82f8e84297d1acce8c32f3e6559849c6cc9bb80da1de82: Status 404 returned error can't find the container with id b36ed3d5bdfb09e96f82f8e84297d1acce8c32f3e6559849c6cc9bb80da1de82
Oct 30 07:22:04.340789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.340751    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:04.340963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.340829    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:04.340963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.340871    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:04.655010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.654970    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerStarted","Data":"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"}
Oct 30 07:22:04.655204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.655045    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerStarted","Data":"b36ed3d5bdfb09e96f82f8e84297d1acce8c32f3e6559849c6cc9bb80da1de82"}
Oct 30 07:22:04.707446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.707381    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podStartSLOduration=1.7073591540000002 podStartE2EDuration="1.707359154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:04.703370952 +0000 UTC m=+1485.136134840" watchObservedRunningTime="2026-10-30 07:22:04.707359154 +0000 UTC m=+1485.140123048"
Oct 30 07:22:04.879328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.876615    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:04.963267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.955694    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:22:04.963267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:04.962367    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:22:05.175746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:05.175663    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:22:05.176100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:05.176071    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:22:05.176784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:05.176766    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:22:05.441577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:05.441492    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:05.867199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:05.867119    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:06.248318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:06.241540    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing" containerID="cri-o://b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f" gracePeriod=25
Oct 30 07:22:06.366316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:06.361567    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing" containerID="cri-o://c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481" gracePeriod=25
Oct 30 07:22:06.670652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:06.670621    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:06.671723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:06.671699    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:22:07.179366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:07.178597    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:07.307228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:07.307188    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:07.434389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:07.434263    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:07.519699 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:07.519659    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:22:07.625708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:07.625671    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:08.996152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:08.996117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:09.106754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.106707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:09.158331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.158279    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:22:09.161872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.161853    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.198334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.188790    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-89664e-dockercfg-ztx9c\""
Oct 30 07:22:09.198334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.188968    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-89664e\""
Oct 30 07:22:09.200455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.200423    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:22:09.478808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.478772    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksxl\" (UniqueName: \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.479187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.479158    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.515784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.515708    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:22:09.520682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.520655    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.521776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.520560    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:22:09.598362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.592728    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.598362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.597747    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.598362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.597880    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msw2\" (UniqueName: \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.598362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.597949    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksxl\" (UniqueName: \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.599143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.599120    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.621814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.621780    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksxl\" (UniqueName: \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\") pod \"payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.699420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.699382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.699633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.699460    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msw2\" (UniqueName: \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.700095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.700073    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.723313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.723273    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msw2\" (UniqueName: \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\") pod \"payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.806662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.806572    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:09.835095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.835052    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:09.847856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:09.847804    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:10.322348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.322317    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:22:10.322687 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:10.322316    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b4bd41_42ee_48cb_9ef4_d1899e3d15ab.slice/crio-9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920 WatchSource:0}: Error finding container 9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920: Status 404 returned error can't find the container with id 9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920
Oct 30 07:22:10.347259 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:10.347231    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799f1cf6_674f_4f2c_a873_fcc11bff17da.slice/crio-1fd9af647b989e590a2c51fbf90f44cc06812bd57469ee624f547531af93ae27 WatchSource:0}: Error finding container 1fd9af647b989e590a2c51fbf90f44cc06812bd57469ee624f547531af93ae27: Status 404 returned error can't find the container with id 1fd9af647b989e590a2c51fbf90f44cc06812bd57469ee624f547531af93ae27
Oct 30 07:22:10.347536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.347515    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:22:10.599884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.599789    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:22:10.620852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.620809    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:22:10.726083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.726050    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" event={"ID":"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab","Type":"ContainerStarted","Data":"2ecbaca7c2d69fd04ff9f83d6051aaefcf37ec9bfdc107785a2a23d0412afcfc"}
Oct 30 07:22:10.726278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.726124    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" event={"ID":"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab","Type":"ContainerStarted","Data":"9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920"}
Oct 30 07:22:10.726278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.726246    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:10.728703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.728671    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" event={"ID":"799f1cf6-674f-4f2c-a873-fcc11bff17da","Type":"ContainerStarted","Data":"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"}
Oct 30 07:22:10.729104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.728726    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" event={"ID":"799f1cf6-674f-4f2c-a873-fcc11bff17da","Type":"ContainerStarted","Data":"1fd9af647b989e590a2c51fbf90f44cc06812bd57469ee624f547531af93ae27"}
Oct 30 07:22:10.729514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.729488    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:10.731136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.731086    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:10.798481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.798415    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" podStartSLOduration=1.798394756 podStartE2EDuration="1.798394756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:10.766967877 +0000 UTC m=+1491.199731769" watchObservedRunningTime="2026-10-30 07:22:10.798394756 +0000 UTC m=+1491.231158650"
Oct 30 07:22:10.813332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.813230    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" podStartSLOduration=1.8132118080000001 podStartE2EDuration="1.813211808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:10.81063882 +0000 UTC m=+1491.243402713" watchObservedRunningTime="2026-10-30 07:22:10.813211808 +0000 UTC m=+1491.245975704"
Oct 30 07:22:10.846680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:10.846640    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:11.447092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:11.445548    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:22:11.447092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:11.445911    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:22:12.009878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:12.009844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:12.031066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:12.031034    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"]
Oct 30 07:22:12.042537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:12.042507    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693bf6-vsnrf"]
Oct 30 07:22:13.435767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:13.435728    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:13.699737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:13.699657    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b72dc59-83c0-49e7-a726-cd931c7ef91f" path="/var/lib/kubelet/pods/6b72dc59-83c0-49e7-a726-cd931c7ef91f/volumes"
Oct 30 07:22:13.990720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:13.990640    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:14.109892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:14.109863    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:14.230316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:14.229497    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:14.668749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:14.668718    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:15.102592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.102366    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:22:15.107263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.107236    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.132203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.126380    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-93de68-openshift-default-dockercfg-hqkpk\""
Oct 30 07:22:15.148358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.144605    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:22:15.173513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.173376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:15.174225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174042    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174282    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174488    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174587    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174728    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnlh\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174819    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.174886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174858    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.175011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.174988    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.175094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.175044    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276375    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276479    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276591    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnlh\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276630    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276665    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276715    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276765    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276834    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.276856    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.277833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.277521    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.282335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.278851    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.282335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.278995    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.306528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.295692    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.343958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.341607    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.343958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.343896    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.360387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.346512    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.361500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.361413    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnlh\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\") pod \"e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") " pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.439710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.438693    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:15.640745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.615484    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" podUID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" containerName="payload-processing" containerID="cri-o://93efc8bb9781fefd7e809c1215447d8a1459bf9ee7de40fb8dc19cc6c5be031d" gracePeriod=25
Oct 30 07:22:15.640745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.621888    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" podUID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" containerName="payload-pre-processing" containerID="cri-o://16e0e8e223f4e633e8dcf978ca546211238b787d13d9698131a5cf86ac5df856" gracePeriod=25
Oct 30 07:22:15.766087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.766051    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:15.819096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.813914    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:22:15.819096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.818419    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:15.819096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.818521    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:15.819096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.818570    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:15.857662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.857625    6738 generic.go:356] "Generic (PLEG): container finished" podID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" containerID="93efc8bb9781fefd7e809c1215447d8a1459bf9ee7de40fb8dc19cc6c5be031d" exitCode=0
Oct 30 07:22:15.857850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.857752    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" event={"ID":"2f0f9430-dce1-454c-9a4c-f4faadbb90ff","Type":"ContainerDied","Data":"93efc8bb9781fefd7e809c1215447d8a1459bf9ee7de40fb8dc19cc6c5be031d"}
Oct 30 07:22:15.859751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.859710    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerStarted","Data":"473c7cb79c5b33dc29008dc017c3d1582d37b659938fb23b1a676f35dbcf836c"}
Oct 30 07:22:15.862440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.862411    6738 generic.go:356] "Generic (PLEG): container finished" podID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" containerID="16e0e8e223f4e633e8dcf978ca546211238b787d13d9698131a5cf86ac5df856" exitCode=0
Oct 30 07:22:15.862738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:15.862628    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" event={"ID":"f2f102f5-3591-4fc2-b341-bf7f3e1df095","Type":"ContainerDied","Data":"16e0e8e223f4e633e8dcf978ca546211238b787d13d9698131a5cf86ac5df856"}
Oct 30 07:22:16.012287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.012220    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:22:16.081829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.081796    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:22:16.136639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.136610    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\") pod \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") "
Oct 30 07:22:16.136889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.136700    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\" (UniqueName: \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\") pod \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\" (UID: \"2f0f9430-dce1-454c-9a4c-f4faadbb90ff\") "
Oct 30 07:22:16.137395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.137360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume" pod "2f0f9430-dce1-454c-9a4c-f4faadbb90ff" (UID: "2f0f9430-dce1-454c-9a4c-f4faadbb90ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:16.137597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.137579    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.139771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.139746    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8" pod "2f0f9430-dce1-454c-9a4c-f4faadbb90ff" (UID: "2f0f9430-dce1-454c-9a4c-f4faadbb90ff"). InnerVolumeSpecName "kube-api-access-pp2x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:16.238760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.238659    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\") pod \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") "
Oct 30 07:22:16.239186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.238829    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\" (UniqueName: \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\") pod \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\" (UID: \"f2f102f5-3591-4fc2-b341-bf7f3e1df095\") "
Oct 30 07:22:16.239186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.239137    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume" pod "f2f102f5-3591-4fc2-b341-bf7f3e1df095" (UID: "f2f102f5-3591-4fc2-b341-bf7f3e1df095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:16.240168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.240143    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2x8\" (UniqueName: \"kubernetes.io/projected/2f0f9430-dce1-454c-9a4c-f4faadbb90ff-kube-api-access-pp2x8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.240273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.240172    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f102f5-3591-4fc2-b341-bf7f3e1df095-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.241679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.241654    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2" pod "f2f102f5-3591-4fc2-b341-bf7f3e1df095" (UID: "f2f102f5-3591-4fc2-b341-bf7f3e1df095"). InnerVolumeSpecName "kube-api-access-9hqw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:16.294750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.294711    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:16.367877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.364070    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hqw2\" (UniqueName: \"kubernetes.io/projected/f2f102f5-3591-4fc2-b341-bf7f3e1df095-kube-api-access-9hqw2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.782529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.782483    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:22:16.783166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.782560    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:22:16.872463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.872433    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"
Oct 30 07:22:16.872652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.872427    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4" event={"ID":"2f0f9430-dce1-454c-9a4c-f4faadbb90ff","Type":"ContainerDied","Data":"c6d14d24ca12c3dc5d86e61368eba1d4b6eb915370d5515dcb5f2c49d4fcc02b"}
Oct 30 07:22:16.872652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.872588    6738 scope.go:122] "RemoveContainer" containerID="93efc8bb9781fefd7e809c1215447d8a1459bf9ee7de40fb8dc19cc6c5be031d"
Oct 30 07:22:16.874438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.874412    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerStarted","Data":"4fa4f12c860e46186b0d8c037b25b425513e020758b674d81c424e948cd2f7ac"}
Oct 30 07:22:16.876033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.876016    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"
Oct 30 07:22:16.876164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.876061    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg" event={"ID":"f2f102f5-3591-4fc2-b341-bf7f3e1df095","Type":"ContainerDied","Data":"4a458d465cbd438839f18762050dc38afadc0f5e478c57d21620eae399866c59"}
Oct 30 07:22:16.885458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.885437    6738 scope.go:122] "RemoveContainer" containerID="16e0e8e223f4e633e8dcf978ca546211238b787d13d9698131a5cf86ac5df856"
Oct 30 07:22:16.936107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.936053    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podStartSLOduration=1.936038612 podStartE2EDuration="1.936038612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:16.93352676 +0000 UTC m=+1497.366290655" watchObservedRunningTime="2026-10-30 07:22:16.936038612 +0000 UTC m=+1497.368802504"
Oct 30 07:22:16.986772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.986740    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:22:16.992579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:16.992548    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b263d4-6cb847fff5-6qrsg"]
Oct 30 07:22:17.026836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.026804    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:22:17.030351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.030322    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b263d4-6976fb566-8x7n4"]
Oct 30 07:22:17.439261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.439220    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:17.445193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.445157    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:17.692705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.692619    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" path="/var/lib/kubelet/pods/2f0f9430-dce1-454c-9a4c-f4faadbb90ff/volumes"
Oct 30 07:22:17.693072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.693055    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" path="/var/lib/kubelet/pods/f2f102f5-3591-4fc2-b341-bf7f3e1df095/volumes"
Oct 30 07:22:17.888612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.888585    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:17.890043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:17.889730    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:22:18.851555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.851503    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:18.940319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.940010    6738 generic.go:356] "Generic (PLEG): container finished" podID="c7979202-6d70-45ed-8ed5-8045828a4633" containerID="a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978" exitCode=0
Oct 30 07:22:18.950476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.941105    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerDied","Data":"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"}
Oct 30 07:22:18.950476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.947514    6738 scope.go:122] "RemoveContainer" containerID="a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"
Oct 30 07:22:18.954421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.954168    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:18.954421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.954280    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:18.954421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:18.954355    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:19.436644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:19.436613    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:19.565039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:19.565001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:19.950561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:19.950521    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerStarted","Data":"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"}
Oct 30 07:22:20.065144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:20.065095    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:22:20.065439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:20.065414    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:20.065552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:20.065473    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:21.065232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.065193    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:21.065683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.065258    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:21.194743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.194703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:21.334428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.334335    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:21.350917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.350881    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:22:21.351759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351733    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" containerName="payload-pre-processing"
Oct 30 07:22:21.351759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351756    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" containerName="payload-pre-processing"
Oct 30 07:22:21.351927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351799    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" containerName="payload-processing"
Oct 30 07:22:21.351927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351807    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" containerName="payload-processing"
Oct 30 07:22:21.351927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351927    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0f9430-dce1-454c-9a4c-f4faadbb90ff" containerName="payload-processing"
Oct 30 07:22:21.352025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.351943    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f102f5-3591-4fc2-b341-bf7f3e1df095" containerName="payload-pre-processing"
Oct 30 07:22:21.355581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.355562    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.400896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.400853    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:22:21.403272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.403243    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-93de68\""
Oct 30 07:22:21.403484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.403419    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-93de68-dockercfg-zd6b6\""
Oct 30 07:22:21.484781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.469592    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc9g\" (UniqueName: \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.484781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.469750    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.510514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.510458    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:22:21.523549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.517917    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:22:21.599369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.575252    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.599369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.575542    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc9g\" (UniqueName: \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.599369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.576787    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.633362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.624659    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:21.730130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.727397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc9g\" (UniqueName: \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\") pod \"payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:21.740550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.737117    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:22:21.740550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.740411    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:22:21.740550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.740500    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:21.800645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.800613    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:22:21.901684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.901646    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cq6\" (UniqueName: \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:21.901858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:21.901708    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.002630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.002595    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:22.004122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.004090    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cq6\" (UniqueName: \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.004281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.004142    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.004948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.004911    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.027915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.027882    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cq6\" (UniqueName: \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\") pod \"payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.062625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.062542    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:22.065583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.065549    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:22.065992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.065608    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:22.311516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.311021    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:22:22.317647 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:22.317618    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773eebc8_9a79_44c4_a833_1afc81e138ef.slice/crio-5de715774fcd250c129577ab1df3c1dcf3de2a022e4e93d238b77fbe4387dd70 WatchSource:0}: Error finding container 5de715774fcd250c129577ab1df3c1dcf3de2a022e4e93d238b77fbe4387dd70: Status 404 returned error can't find the container with id 5de715774fcd250c129577ab1df3c1dcf3de2a022e4e93d238b77fbe4387dd70
Oct 30 07:22:22.460093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.460058    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:22:22.460624 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:22.460596    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca713360_0759_445e_88fa_5bf0e0014adb.slice/crio-7cceff3d753b23a91f7efbee4b4cafbdf407fab75f2f41a55f4472d0e74a5430 WatchSource:0}: Error finding container 7cceff3d753b23a91f7efbee4b4cafbdf407fab75f2f41a55f4472d0e74a5430: Status 404 returned error can't find the container with id 7cceff3d753b23a91f7efbee4b4cafbdf407fab75f2f41a55f4472d0e74a5430
Oct 30 07:22:22.803933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:22.803848    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:22:23.016777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.016738    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" event={"ID":"ca713360-0759-445e-88fa-5bf0e0014adb","Type":"ContainerStarted","Data":"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"}
Oct 30 07:22:23.017008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.016802    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" event={"ID":"ca713360-0759-445e-88fa-5bf0e0014adb","Type":"ContainerStarted","Data":"7cceff3d753b23a91f7efbee4b4cafbdf407fab75f2f41a55f4472d0e74a5430"}
Oct 30 07:22:23.017008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.016832    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:23.018769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.018739    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" event={"ID":"773eebc8-9a79-44c4-a833-1afc81e138ef","Type":"ContainerStarted","Data":"84a5e860e3fbb22db61c41fc58c30ddf1a590809c7f24fe7d81f4d59d39a69c8"}
Oct 30 07:22:23.018911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.018786    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" event={"ID":"773eebc8-9a79-44c4-a833-1afc81e138ef","Type":"ContainerStarted","Data":"5de715774fcd250c129577ab1df3c1dcf3de2a022e4e93d238b77fbe4387dd70"}
Oct 30 07:22:23.018911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.018834    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:23.060940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.060813    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" podStartSLOduration=2.06079652 podStartE2EDuration="2.06079652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:23.058194887 +0000 UTC m=+1503.490958780" watchObservedRunningTime="2026-10-30 07:22:23.06079652 +0000 UTC m=+1503.493560413"
Oct 30 07:22:23.065687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.065651    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:23.066114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.065704    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:23.117121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.117058    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" podStartSLOduration=2.117041521 podStartE2EDuration="2.117041521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:23.114545059 +0000 UTC m=+1503.547308968" watchObservedRunningTime="2026-10-30 07:22:23.117041521 +0000 UTC m=+1503.549805411"
Oct 30 07:22:23.796313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:23.796263    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:24.065274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:24.065184    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:24.065274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:24.065251    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:24.310379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:24.310342    6738 scope.go:122] "RemoveContainer" containerID="b8bdc9699efcc5135ef3c059f0b14353df6940e1169606c5b309c13ac6e831a8"
Oct 30 07:22:24.320911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:24.320882    6738 scope.go:122] "RemoveContainer" containerID="371d53bed1013879ecc340eb92edbfab1e49bd1b6de331dae0595e7b8ebda504"
Oct 30 07:22:25.065737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.065702    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:22:25.065944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.065755    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 07:22:25.499648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.499616    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"]
Oct 30 07:22:25.504459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.504437    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.522954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.522919    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"]
Oct 30 07:22:25.530914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.530887    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:22:25.532251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.532225    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:22:25.532459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.532232    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:22:25.532616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.532251    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:22:25.600345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.600281    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5m9\" (UniqueName: \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.600544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.600461    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.600685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.600670    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.702585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.702550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.702585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.702603    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5m9\" (UniqueName: \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.702867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.702708    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.703283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.703260    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.705355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.705326    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.726457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.726426    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5m9\" (UniqueName: \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\") pod \"maas-api-revoke-sub-ba1c4231-f844s\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.820394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.820304    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:25.863173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:25.863134    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:26.069679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.069643    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:22:26.070121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.070099    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:22:26.070850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.070792    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:22:26.171052 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:26.171020    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967f22d4_146a_49a2_b045_d5fa3b42fb26.slice/crio-fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa WatchSource:0}: Error finding container fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa: Status 404 returned error can't find the container with id fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa
Oct 30 07:22:26.173595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.173571    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"]
Oct 30 07:22:26.509161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.509129    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:22:26.644918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.644655    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\") pod \"213d2a9c-fee7-401d-acc5-df917039a58a\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") "
Oct 30 07:22:26.644918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.644817    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\" (UniqueName: \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\") pod \"213d2a9c-fee7-401d-acc5-df917039a58a\" (UID: \"213d2a9c-fee7-401d-acc5-df917039a58a\") "
Oct 30 07:22:26.646691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.646415    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume" pod "213d2a9c-fee7-401d-acc5-df917039a58a" (UID: "213d2a9c-fee7-401d-acc5-df917039a58a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:26.648887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.648863    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp" pod "213d2a9c-fee7-401d-acc5-df917039a58a" (UID: "213d2a9c-fee7-401d-acc5-df917039a58a"). InnerVolumeSpecName "kube-api-access-sv9qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:26.650673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.650651    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:22:26.747076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.747040    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\" (UniqueName: \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\") pod \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") "
Oct 30 07:22:26.747277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.747097    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\") pod \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\" (UID: \"397d80c5-c41d-46a2-b09f-db22e8cfec3f\") "
Oct 30 07:22:26.747753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.747727    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213d2a9c-fee7-401d-acc5-df917039a58a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:26.747753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.747752    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9qp\" (UniqueName: \"kubernetes.io/projected/213d2a9c-fee7-401d-acc5-df917039a58a-kube-api-access-sv9qp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:26.748386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.748360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume" pod "397d80c5-c41d-46a2-b09f-db22e8cfec3f" (UID: "397d80c5-c41d-46a2-b09f-db22e8cfec3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:26.750347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.750324    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm" pod "397d80c5-c41d-46a2-b09f-db22e8cfec3f" (UID: "397d80c5-c41d-46a2-b09f-db22e8cfec3f"). InnerVolumeSpecName "kube-api-access-9nhvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:26.849042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.849000    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhvm\" (UniqueName: \"kubernetes.io/projected/397d80c5-c41d-46a2-b09f-db22e8cfec3f-kube-api-access-9nhvm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:26.849042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:26.849037    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397d80c5-c41d-46a2-b09f-db22e8cfec3f-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:27.050472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.050376    6738 generic.go:356] "Generic (PLEG): container finished" podID="213d2a9c-fee7-401d-acc5-df917039a58a" containerID="b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f" exitCode=0
Oct 30 07:22:27.050472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.050426    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"
Oct 30 07:22:27.050717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.050464    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" event={"ID":"213d2a9c-fee7-401d-acc5-df917039a58a","Type":"ContainerDied","Data":"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"}
Oct 30 07:22:27.050717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.050525    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg" event={"ID":"213d2a9c-fee7-401d-acc5-df917039a58a","Type":"ContainerDied","Data":"4719dbbddf4ad51e3292b095a633c5e6472a8994e9b432dbce54bd6a1828b664"}
Oct 30 07:22:27.050717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.050553    6738 scope.go:122] "RemoveContainer" containerID="b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"
Oct 30 07:22:27.052472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.052444    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s" event={"ID":"967f22d4-146a-49a2-b045-d5fa3b42fb26","Type":"ContainerStarted","Data":"59b0d9cc6ee3f399c2e67f4107ee27d13f50acdad0169a63e884286702e93aed"}
Oct 30 07:22:27.052624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.052495    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s" event={"ID":"967f22d4-146a-49a2-b045-d5fa3b42fb26","Type":"ContainerStarted","Data":"fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa"}
Oct 30 07:22:27.054097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.054071    6738 generic.go:356] "Generic (PLEG): container finished" podID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerID="c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481" exitCode=0
Oct 30 07:22:27.054214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.054115    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"
Oct 30 07:22:27.054214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.054120    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" event={"ID":"397d80c5-c41d-46a2-b09f-db22e8cfec3f","Type":"ContainerDied","Data":"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"}
Oct 30 07:22:27.054214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.054164    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q" event={"ID":"397d80c5-c41d-46a2-b09f-db22e8cfec3f","Type":"ContainerDied","Data":"355d5db40da5e7652188eddc9e074a290976bb4aef0de96666b250794534a73e"}
Oct 30 07:22:27.064105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.064063    6738 scope.go:122] "RemoveContainer" containerID="b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"
Oct 30 07:22:27.064434 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:27.064415    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f\": container with ID starting with b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f not found: ID does not exist" containerID="b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"
Oct 30 07:22:27.064518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.064442    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f"} err="failed to get container status \"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f\": rpc error: code = NotFound desc = could not find container \"b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f\": container with ID starting with b446e15b46cc408a46572280a16343fb8205bacc7427f23f24ad08c7d356747f not found: ID does not exist"
Oct 30 07:22:27.064518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.064461    6738 scope.go:122] "RemoveContainer" containerID="c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"
Oct 30 07:22:27.079388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.079356    6738 scope.go:122] "RemoveContainer" containerID="c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"
Oct 30 07:22:27.079974 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:27.079953    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481\": container with ID starting with c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481 not found: ID does not exist" containerID="c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"
Oct 30 07:22:27.080060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.079985    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481"} err="failed to get container status \"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481\": rpc error: code = NotFound desc = could not find container \"c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481\": container with ID starting with c88b868b9463ba53166c70cbb4ce73c7fd99ff478c4db73e9196f397ffb0a481 not found: ID does not exist"
Oct 30 07:22:27.104406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.104372    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:22:27.116415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.116382    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a51baa-765479dcf-hmd8q"]
Oct 30 07:22:27.155334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.155084    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s" podStartSLOduration=2.155048063 podStartE2EDuration="2.155048063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:27.14971618 +0000 UTC m=+1507.582480086" watchObservedRunningTime="2026-10-30 07:22:27.155048063 +0000 UTC m=+1507.587811956"
Oct 30 07:22:27.187833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.187800    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:22:27.192879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.192853    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a51baa-7b64cbb66-ltbwg"]
Oct 30 07:22:27.691809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.691775    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" path="/var/lib/kubelet/pods/213d2a9c-fee7-401d-acc5-df917039a58a/volumes"
Oct 30 07:22:27.692279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.692229    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" path="/var/lib/kubelet/pods/397d80c5-c41d-46a2-b09f-db22e8cfec3f/volumes"
Oct 30 07:22:27.695496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:27.695473    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:28.064383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:28.064252    6738 generic.go:356] "Generic (PLEG): container finished" podID="967f22d4-146a-49a2-b045-d5fa3b42fb26" containerID="59b0d9cc6ee3f399c2e67f4107ee27d13f50acdad0169a63e884286702e93aed" exitCode=0
Oct 30 07:22:28.064383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:28.064346    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s" event={"ID":"967f22d4-146a-49a2-b045-d5fa3b42fb26","Type":"ContainerDied","Data":"59b0d9cc6ee3f399c2e67f4107ee27d13f50acdad0169a63e884286702e93aed"}
Oct 30 07:22:29.233979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.233948    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:29.397197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.397146    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\" (UniqueName: \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\") pod \"967f22d4-146a-49a2-b045-d5fa3b42fb26\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") "
Oct 30 07:22:29.397463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.397333    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\") pod \"967f22d4-146a-49a2-b045-d5fa3b42fb26\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") "
Oct 30 07:22:29.397463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.397423    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\") pod \"967f22d4-146a-49a2-b045-d5fa3b42fb26\" (UID: \"967f22d4-146a-49a2-b045-d5fa3b42fb26\") "
Oct 30 07:22:29.397706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.397674    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca" pod "967f22d4-146a-49a2-b045-d5fa3b42fb26" (UID: "967f22d4-146a-49a2-b045-d5fa3b42fb26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:29.398121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.398101    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:29.399515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.399495    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca" pod "967f22d4-146a-49a2-b045-d5fa3b42fb26" (UID: "967f22d4-146a-49a2-b045-d5fa3b42fb26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:22:29.399602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.399571    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9" pod "967f22d4-146a-49a2-b045-d5fa3b42fb26" (UID: "967f22d4-146a-49a2-b045-d5fa3b42fb26"). InnerVolumeSpecName "kube-api-access-9g5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:29.499364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.499285    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967f22d4-146a-49a2-b045-d5fa3b42fb26-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:29.499364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.499361    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5m9\" (UniqueName: \"kubernetes.io/projected/967f22d4-146a-49a2-b045-d5fa3b42fb26-kube-api-access-9g5m9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:29.521907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:29.521862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:30.081518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:30.081425    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"
Oct 30 07:22:30.081518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:30.081454    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s" event={"ID":"967f22d4-146a-49a2-b045-d5fa3b42fb26","Type":"ContainerDied","Data":"fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa"}
Oct 30 07:22:30.081518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:30.081513    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb65a3eb5e0665deeb473f5796d1f8fed42041924917b9abfa80b4f0d20bb4aa"
Oct 30 07:22:30.179559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:30.179514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:31.157419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:31.157387    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:31.816148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:31.816101    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:31.924587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:31.924551    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"]
Oct 30 07:22:31.936726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:31.936688    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b263d4-kw7rj"]
Oct 30 07:22:31.953247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:31.953214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:32.068258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:32.068162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:32.485242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:32.485207    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:33.691908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:33.691874    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b04ea9-2a60-46fc-b988-d8f0f2912578" path="/var/lib/kubelet/pods/35b04ea9-2a60-46fc-b988-d8f0f2912578/volumes"
Oct 30 07:22:33.917453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:33.917413    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:34.042384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:34.039864    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:22:34.042384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:34.041822    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:22:34.049988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:34.049904    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:34.168960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:34.168922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:35.958935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:35.958902    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:36.169556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:36.169516    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:36.316586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:36.316502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:37.523968 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:37.523932    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:22:38.038021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:38.037978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:39.661927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:39.661895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:39.802244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:39.802202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:39.916919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:39.916834    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:22:39.917355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:39.917321    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" containerID="cri-o://7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465" gracePeriod=30
Oct 30 07:22:39.992720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:39.992678    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:40.251016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.250928    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:40.558965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.558882    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"]
Oct 30 07:22:40.559590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559575    6738 cpu_manager.go:414] "containerMap: removing container" podUID="967f22d4-146a-49a2-b045-d5fa3b42fb26" containerName="revoke-keys"
Oct 30 07:22:40.559646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559591    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967f22d4-146a-49a2-b045-d5fa3b42fb26" containerName="revoke-keys"
Oct 30 07:22:40.559646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559619    6738 cpu_manager.go:414] "containerMap: removing container" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing"
Oct 30 07:22:40.559646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559624    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing"
Oct 30 07:22:40.559646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559644    6738 cpu_manager.go:414] "containerMap: removing container" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing"
Oct 30 07:22:40.559781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559648    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing"
Oct 30 07:22:40.559781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559773    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="397d80c5-c41d-46a2-b09f-db22e8cfec3f" containerName="payload-pre-processing"
Oct 30 07:22:40.559849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559784    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="213d2a9c-fee7-401d-acc5-df917039a58a" containerName="payload-processing"
Oct 30 07:22:40.559849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.559795    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="967f22d4-146a-49a2-b045-d5fa3b42fb26" containerName="revoke-keys"
Oct 30 07:22:40.562699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.562677    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.598139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.598101    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:22:40.598501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.598475    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:22:40.602318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.601157    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"]
Oct 30 07:22:40.636678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.636638    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:22:40.636863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.636814    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:22:40.738246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.738206    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.738874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.738847    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.745505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.739096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7p9\" (UniqueName: \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.841356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.841246    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.841356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.841310    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.841356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.841343    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7p9\" (UniqueName: \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.842115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.842086    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.843827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.843802    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.866007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.865978    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7p9\" (UniqueName: \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\") pod \"maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:40.918765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:40.918732    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:41.243713 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:41.243673    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bef4f20_6702_434d_81b1_f3e993f75379.slice/crio-6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02 WatchSource:0}: Error finding container 6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02: Status 404 returned error can't find the container with id 6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02
Oct 30 07:22:41.244061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:41.244037    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"]
Oct 30 07:22:42.233362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:42.233326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8" event={"ID":"3bef4f20-6702-434d-81b1-f3e993f75379","Type":"ContainerStarted","Data":"51b19940eee9bfbf2f6163274a0f8d2cb6bc3487d593b1b65968d96191e48059"}
Oct 30 07:22:42.233885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:42.233390    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8" event={"ID":"3bef4f20-6702-434d-81b1-f3e993f75379","Type":"ContainerStarted","Data":"6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02"}
Oct 30 07:22:42.295849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:42.295784    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8" podStartSLOduration=2.295762512 podStartE2EDuration="2.295762512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:42.293482123 +0000 UTC m=+1522.726246018" watchObservedRunningTime="2026-10-30 07:22:42.295762512 +0000 UTC m=+1522.728526409"
Oct 30 07:22:42.439228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:42.439194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:43.242573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:43.242536    6738 generic.go:356] "Generic (PLEG): container finished" podID="3bef4f20-6702-434d-81b1-f3e993f75379" containerID="51b19940eee9bfbf2f6163274a0f8d2cb6bc3487d593b1b65968d96191e48059" exitCode=0
Oct 30 07:22:43.242978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:43.242605    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8" event={"ID":"3bef4f20-6702-434d-81b1-f3e993f75379","Type":"ContainerDied","Data":"51b19940eee9bfbf2f6163274a0f8d2cb6bc3487d593b1b65968d96191e48059"}
Oct 30 07:22:44.586595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.586512    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:44.624514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.624464    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\") pod \"3bef4f20-6702-434d-81b1-f3e993f75379\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") "
Oct 30 07:22:44.624734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.624549    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\") pod \"3bef4f20-6702-434d-81b1-f3e993f75379\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") "
Oct 30 07:22:44.624734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.624598    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\" (UniqueName: \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\") pod \"3bef4f20-6702-434d-81b1-f3e993f75379\" (UID: \"3bef4f20-6702-434d-81b1-f3e993f75379\") "
Oct 30 07:22:44.624955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.624928    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca" pod "3bef4f20-6702-434d-81b1-f3e993f75379" (UID: "3bef4f20-6702-434d-81b1-f3e993f75379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:44.625423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.625397    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:44.626764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.626736    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca" pod "3bef4f20-6702-434d-81b1-f3e993f75379" (UID: "3bef4f20-6702-434d-81b1-f3e993f75379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:22:44.626857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.626812    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9" pod "3bef4f20-6702-434d-81b1-f3e993f75379" (UID: "3bef4f20-6702-434d-81b1-f3e993f75379"). InnerVolumeSpecName "kube-api-access-qc7p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:44.672186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.672151    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:44.673644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.673613    6738 patch_prober.go:29] interesting pod/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:22:44.673814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.673679    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:22:44.757206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.754709    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bef4f20-6702-434d-81b1-f3e993f75379-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:44.757206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:44.754743    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7p9\" (UniqueName: \"kubernetes.io/projected/3bef4f20-6702-434d-81b1-f3e993f75379-kube-api-access-qc7p9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.181149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.181079    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:22:45.263167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263134    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263199    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263307    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263425    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263464    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263517    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:45.263638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263627    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263687    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263731    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.263822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.263808    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\") pod \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\" (UID: \"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c\") "
Oct 30 07:22:45.264579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.264558    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.264889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.264866    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:45.265081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.265062    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:45.265378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.265353    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:45.265669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.265643    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:45.267572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.267533    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:45.267862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.267838    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:45.267935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.267895    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:22:45.268133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.268114    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd" pod "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" (UID: "a9d550f8-5ef4-473e-a0e4-4d005b40ee8c"). InnerVolumeSpecName "kube-api-access-nsvhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:45.308618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.308580    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"
Oct 30 07:22:45.308618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.308602    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8" event={"ID":"3bef4f20-6702-434d-81b1-f3e993f75379","Type":"ContainerDied","Data":"6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02"}
Oct 30 07:22:45.308845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.308682    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6342b7bac16171db40647f69c5d6203231d232d08f99bc184089c23a03047f02"
Oct 30 07:22:45.310648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.310619    6738 generic.go:356] "Generic (PLEG): container finished" podID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerID="7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465" exitCode=0
Oct 30 07:22:45.310795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.310669    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"
Oct 30 07:22:45.310795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.310703    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerDied","Data":"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"}
Oct 30 07:22:45.310795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.310758    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4" event={"ID":"a9d550f8-5ef4-473e-a0e4-4d005b40ee8c","Type":"ContainerDied","Data":"a3ea42fd0fd8c8513cbf0ad5573e136adf7365bcc8bcc144ba7e0c1ab3f595ed"}
Oct 30 07:22:45.310795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.310783    6738 scope.go:122] "RemoveContainer" containerID="7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"
Oct 30 07:22:45.323848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.323669    6738 scope.go:122] "RemoveContainer" containerID="94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"
Oct 30 07:22:45.335382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.335349    6738 scope.go:122] "RemoveContainer" containerID="7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"
Oct 30 07:22:45.335788 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:45.335748    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465\": container with ID starting with 7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465 not found: ID does not exist" containerID="7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"
Oct 30 07:22:45.335881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.335801    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465"} err="failed to get container status \"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465\": rpc error: code = NotFound desc = could not find container \"7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465\": container with ID starting with 7bfaabfdbc0d2c7fb59c0f07050b380be45ebe0afee8f11de9ecb0d94b248465 not found: ID does not exist"
Oct 30 07:22:45.335881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.335829    6738 scope.go:122] "RemoveContainer" containerID="94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"
Oct 30 07:22:45.336135 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:22:45.336111    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f\": container with ID starting with 94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f not found: ID does not exist" containerID="94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"
Oct 30 07:22:45.336194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.336142    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f"} err="failed to get container status \"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f\": rpc error: code = NotFound desc = could not find container \"94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f\": container with ID starting with 94c16f63e81a88cfb318387b69ea8ec22c9265b6991a4cbddad48fbb5084bd7f not found: ID does not exist"
Oct 30 07:22:45.366114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366074    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366115    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366132    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366147    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366160    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366173    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366187    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.366335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.366203    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsvhd\" (UniqueName: \"kubernetes.io/projected/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c-kube-api-access-nsvhd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:45.376923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.376891    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:22:45.382104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.382078    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b263d4-openshift-default-c487694bf-zw2b4"]
Oct 30 07:22:45.691968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:45.691935    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" path="/var/lib/kubelet/pods/a9d550f8-5ef4-473e-a0e4-4d005b40ee8c/volumes"
Oct 30 07:22:46.035595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:46.035557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:47.220569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.220531    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:22:47.221472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221449    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3bef4f20-6702-434d-81b1-f3e993f75379" containerName="revoke-keys"
Oct 30 07:22:47.221597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221474    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bef4f20-6702-434d-81b1-f3e993f75379" containerName="revoke-keys"
Oct 30 07:22:47.221597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221526    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.221597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221535    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.221767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221718    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bef4f20-6702-434d-81b1-f3e993f75379" containerName="revoke-keys"
Oct 30 07:22:47.221767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221750    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.221767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.221764    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.225493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.225472    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.251946    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-93de68-serving-cert\""
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.256450    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.258168    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.258186    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnjcd\""
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.258246    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:22:47.262962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.258316    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:22:47.294941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.294906    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.295124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.295047    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k66vk\" (UniqueName: \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.295177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.295127    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.295211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.295179    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.306206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.306177    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:47.396747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.396713    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.396960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.396810    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k66vk\" (UniqueName: \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.396960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.396859    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.396960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.396908    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.397943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.397825    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.399791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.399762    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.403806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.403778    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.423749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.423710    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k66vk\" (UniqueName: \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\") pod \"maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.438522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.438484    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:47.541259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.541168    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:47.552240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.552198    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:47.679579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.679544    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"]
Oct 30 07:22:47.680259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.680243    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.680341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.680263    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d550f8-5ef4-473e-a0e4-4d005b40ee8c" containerName="istio-proxy"
Oct 30 07:22:47.683306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.683262    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.702431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.702394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcgf\" (UniqueName: \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.702788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.702755    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.702908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.702884    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.703746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.703684    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:22:47.703746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.703709    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:22:47.710621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.710589    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"]
Oct 30 07:22:47.805811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.805171    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcgf\" (UniqueName: \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.805811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.805263    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.805811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.805423    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.806418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.806390    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.807954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.807929    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.828913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.828878    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcgf\" (UniqueName: \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:47.854076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.854042    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:22:47.854076 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:47.854058    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a81f0c8_77b3_45ed_9084_b8464e21d512.slice/crio-5880e20cd27dbc05c47f813e0dee376096da607861631b1be86cc49e5d277d6a WatchSource:0}: Error finding container 5880e20cd27dbc05c47f813e0dee376096da607861631b1be86cc49e5d277d6a: Status 404 returned error can't find the container with id 5880e20cd27dbc05c47f813e0dee376096da607861631b1be86cc49e5d277d6a
Oct 30 07:22:47.856497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:47.856195    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:22:48.010774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:48.010736    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:48.317872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:48.317792    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"]
Oct 30 07:22:48.318367 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:22:48.317984    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f1d4ff_94d7_4c96_afc7_8450e6cb9eca.slice/crio-49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa WatchSource:0}: Error finding container 49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa: Status 404 returned error can't find the container with id 49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa
Oct 30 07:22:48.339868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:48.339809    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2" event={"ID":"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca","Type":"ContainerStarted","Data":"49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa"}
Oct 30 07:22:48.341389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:48.341363    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" event={"ID":"7a81f0c8-77b3-45ed-9084-b8464e21d512","Type":"ContainerStarted","Data":"5880e20cd27dbc05c47f813e0dee376096da607861631b1be86cc49e5d277d6a"}
Oct 30 07:22:49.354267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.354228    6738 generic.go:356] "Generic (PLEG): container finished" podID="65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" containerID="8904a40b968ef684ad77a86dcb69d641b92ffb35d49f9e8e9962f77f8093e531" exitCode=0
Oct 30 07:22:49.354772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.354324    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2" event={"ID":"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca","Type":"ContainerDied","Data":"8904a40b968ef684ad77a86dcb69d641b92ffb35d49f9e8e9962f77f8093e531"}
Oct 30 07:22:49.356480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.356456    6738 generic.go:356] "Generic (PLEG): container finished" podID="63a45d26-7fac-475b-91a4-c305f67aa821" containerID="f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f" exitCode=0
Oct 30 07:22:49.356627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.356537    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerDied","Data":"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"}
Oct 30 07:22:49.357024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.357005    6738 scope.go:122] "RemoveContainer" containerID="f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"
Oct 30 07:22:49.359059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.359024    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:49.359171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.359118    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:49.359222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.359170    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:22:49.531489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:49.531400    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:50.366971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.366875    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" event={"ID":"7a81f0c8-77b3-45ed-9084-b8464e21d512","Type":"ContainerStarted","Data":"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"}
Oct 30 07:22:50.366971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.366945    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:50.389317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.386270    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerStarted","Data":"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"}
Oct 30 07:22:50.413662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.413603    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" podStartSLOduration=1.329782466 podStartE2EDuration="3.413587253s" totalImagesPullingTime="2.083804787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:50.411590915 +0000 UTC m=+1530.844354809" watchObservedRunningTime="2026-10-30 07:22:50.413587253 +0000 UTC m=+1530.846351207"
Oct 30 07:22:50.656399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.656371    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:50.824200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.824167    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\") pod \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") "
Oct 30 07:22:50.824389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.824217    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\" (UniqueName: \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\") pod \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") "
Oct 30 07:22:50.824389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.824328    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\") pod \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\" (UID: \"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca\") "
Oct 30 07:22:50.824742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.824710    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca" pod "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" (UID: "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:50.825111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.825092    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:50.826412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.826380    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca" pod "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" (UID: "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:22:50.826506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.826406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf" pod "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" (UID: "65f1d4ff-94d7-4c96-afc7-8450e6cb9eca"). InnerVolumeSpecName "kube-api-access-2jcgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:50.926525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.926442    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:50.926525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:50.926474    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcgf\" (UniqueName: \"kubernetes.io/projected/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca-kube-api-access-2jcgf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:51.098401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.098365    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:22:51.136104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.136072    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:22:51.195376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.186729    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:22:51.200507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.199341    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 07:22:51.200507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.199412    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 07:22:51.261813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.261773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:51.387316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.386731    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:51.453575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.452333    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"
Oct 30 07:22:51.453575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.452551    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2" event={"ID":"65f1d4ff-94d7-4c96-afc7-8450e6cb9eca","Type":"ContainerDied","Data":"49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa"}
Oct 30 07:22:51.453575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.452631    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49364ced126ae3e64dae01edfe06546d3978a6e9d563a8db7116c31f0bf9c8fa"
Oct 30 07:22:51.793381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:51.791400    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:52.182221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:52.181751    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 07:22:52.182221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:52.181819    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 07:22:53.188317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:53.187752    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 07:22:53.188317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:53.187830    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 07:22:53.495273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:53.495173    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:54.177318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.176545    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 07:22:54.177318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.176608    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 07:22:54.192885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.192841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:54.310713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.310672    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:54.350680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.350651    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"]
Oct 30 07:22:54.363081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.363031    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-704bf6-m5mv8"]
Oct 30 07:22:54.473382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.467428    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:54.832911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:54.832808    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:55.190319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:55.189445    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 07:22:55.190319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:55.189514    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 07:22:55.690993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:55.690960    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bef4f20-6702-434d-81b1-f3e993f75379" path="/var/lib/kubelet/pods/3bef4f20-6702-434d-81b1-f3e993f75379/volumes"
Oct 30 07:22:56.099335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.099206    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" podUID="105f610a-3b07-4433-807a-b851ad06e842" containerName="payload-processing" containerID="cri-o://1f36aca6b883caf80f979683c99a5d879d4f7a89c8bb1e2445ba0ecb440c12fc" gracePeriod=25
Oct 30 07:22:56.136410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.136372    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" podUID="63074c69-da63-414a-8123-00065b48db97" containerName="payload-pre-processing" containerID="cri-o://7f1e73be2cf6b58b14b3d6f4d17a1f55ef4c7a917d9d67a307a815952d98d80d" gracePeriod=25
Oct 30 07:22:56.180474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.180448    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:22:56.180888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.180868    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:22:56.181734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.181712    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:22:56.371142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.371107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:56.480510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.480480    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:22:56.510572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.510534    6738 generic.go:356] "Generic (PLEG): container finished" podID="105f610a-3b07-4433-807a-b851ad06e842" containerID="1f36aca6b883caf80f979683c99a5d879d4f7a89c8bb1e2445ba0ecb440c12fc" exitCode=0
Oct 30 07:22:56.510721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.510599    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" event={"ID":"105f610a-3b07-4433-807a-b851ad06e842","Type":"ContainerDied","Data":"1f36aca6b883caf80f979683c99a5d879d4f7a89c8bb1e2445ba0ecb440c12fc"}
Oct 30 07:22:56.510721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.510650    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w" event={"ID":"105f610a-3b07-4433-807a-b851ad06e842","Type":"ContainerDied","Data":"749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e"}
Oct 30 07:22:56.510721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.510673    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="749ca72e7a6ec0ee56289afc4688abec4cb2749b596feebd75dd6c071065033e"
Oct 30 07:22:56.512193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.512171    6738 generic.go:356] "Generic (PLEG): container finished" podID="63074c69-da63-414a-8123-00065b48db97" containerID="7f1e73be2cf6b58b14b3d6f4d17a1f55ef4c7a917d9d67a307a815952d98d80d" exitCode=0
Oct 30 07:22:56.512343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.512245    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" event={"ID":"63074c69-da63-414a-8123-00065b48db97","Type":"ContainerDied","Data":"7f1e73be2cf6b58b14b3d6f4d17a1f55ef4c7a917d9d67a307a815952d98d80d"}
Oct 30 07:22:56.539144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.539115    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:22:56.544501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.544473    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:22:56.621880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.621852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\" (UniqueName: \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\") pod \"63074c69-da63-414a-8123-00065b48db97\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") "
Oct 30 07:22:56.622074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.621992    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\") pod \"63074c69-da63-414a-8123-00065b48db97\" (UID: \"63074c69-da63-414a-8123-00065b48db97\") "
Oct 30 07:22:56.622174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.622155    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\") pod \"105f610a-3b07-4433-807a-b851ad06e842\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") "
Oct 30 07:22:56.622280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.622262    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\" (UniqueName: \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\") pod \"105f610a-3b07-4433-807a-b851ad06e842\" (UID: \"105f610a-3b07-4433-807a-b851ad06e842\") "
Oct 30 07:22:56.622554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.622532    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume" pod "63074c69-da63-414a-8123-00065b48db97" (UID: "63074c69-da63-414a-8123-00065b48db97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:56.622686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.622657    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume" pod "105f610a-3b07-4433-807a-b851ad06e842" (UID: "105f610a-3b07-4433-807a-b851ad06e842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:56.623126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.623102    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f610a-3b07-4433-807a-b851ad06e842-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:56.623230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.623131    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63074c69-da63-414a-8123-00065b48db97-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:56.625033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.625007    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v" pod "105f610a-3b07-4433-807a-b851ad06e842" (UID: "105f610a-3b07-4433-807a-b851ad06e842"). InnerVolumeSpecName "kube-api-access-wwb5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:56.625145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.625132    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5" pod "63074c69-da63-414a-8123-00065b48db97" (UID: "63074c69-da63-414a-8123-00065b48db97"). InnerVolumeSpecName "kube-api-access-2s2v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:56.724738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.724702    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwb5v\" (UniqueName: \"kubernetes.io/projected/105f610a-3b07-4433-807a-b851ad06e842-kube-api-access-wwb5v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:56.724738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:56.724731    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2v5\" (UniqueName: \"kubernetes.io/projected/63074c69-da63-414a-8123-00065b48db97-kube-api-access-2s2v5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:22:57.522085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.522048    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"
Oct 30 07:22:57.522085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.522068    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6" event={"ID":"63074c69-da63-414a-8123-00065b48db97","Type":"ContainerDied","Data":"3ce24cd892695168fcf8f37f7afa0c3fc4ee970cb237842b9fe0a931ee02c255"}
Oct 30 07:22:57.522431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.522126    6738 scope.go:122] "RemoveContainer" containerID="7f1e73be2cf6b58b14b3d6f4d17a1f55ef4c7a917d9d67a307a815952d98d80d"
Oct 30 07:22:57.522431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.522184    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"
Oct 30 07:22:57.591069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.591030    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:22:57.604821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.604787    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-704bf6-5794b9c756-s8d9w"]
Oct 30 07:22:57.630320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.630276    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:22:57.640659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.640630    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-704bf6-57789878d7-flqk6"]
Oct 30 07:22:57.692008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.691971    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105f610a-3b07-4433-807a-b851ad06e842" path="/var/lib/kubelet/pods/105f610a-3b07-4433-807a-b851ad06e842/volumes"
Oct 30 07:22:57.692486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:57.692472    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63074c69-da63-414a-8123-00065b48db97" path="/var/lib/kubelet/pods/63074c69-da63-414a-8123-00065b48db97/volumes"
Oct 30 07:22:58.052306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.052252    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:58.154379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.154335    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:22:58.174179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.174136    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:22:58.243631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.243596    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:58.359336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.359215    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:58.499829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:58.499793    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.216005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.215924    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.330159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.330106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.365553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.365519    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"]
Oct 30 07:22:59.377791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.377748    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a51baa-4htkr"]
Oct 30 07:22:59.448588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.445632    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.600564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.600467    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.692116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.692083    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4299cf55-9fa2-4530-a6bd-5b99026b7b25" path="/var/lib/kubelet/pods/4299cf55-9fa2-4530-a6bd-5b99026b7b25/volumes"
Oct 30 07:22:59.717216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.717181    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:22:59.788686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.788651    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"]
Oct 30 07:22:59.797881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.797841    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2504c2-mtdj2"]
Oct 30 07:22:59.919934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:22:59.919897    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:00.248372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:00.243952    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:00.396886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:00.396842    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:23:00.397142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:00.397115    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy" containerID="cri-o://cbf2078ef86c10ae5cdfcf32a2e28a4f0199e45154b1d67d497d35b3c539d6ff" gracePeriod=30
Oct 30 07:23:01.695097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:01.695022    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" path="/var/lib/kubelet/pods/65f1d4ff-94d7-4c96-afc7-8450e6cb9eca/volumes"
Oct 30 07:23:01.849910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:01.849869    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:01.993486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:01.993380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:02.206585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:02.206543    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:23:02.206913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:02.206879    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" containerID="cri-o://be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330" gracePeriod=30
Oct 30 07:23:02.210401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:02.210374    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:23:02.210664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:02.210616    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" containerID="cri-o://8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b" gracePeriod=30
Oct 30 07:23:02.233661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:02.233629    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:03.155440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.155370    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" podUID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" containerName="payload-processing" containerID="cri-o://12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce" gracePeriod=25
Oct 30 07:23:03.188371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.181387    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" podUID="87305586-cd7e-4814-bd76-09c27728dafc" containerName="payload-pre-processing" containerID="cri-o://646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3" gracePeriod=25
Oct 30 07:23:03.541639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.541572    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:23:03.565534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.565507    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:23:03.611803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.611758    6738 generic.go:356] "Generic (PLEG): container finished" podID="87305586-cd7e-4814-bd76-09c27728dafc" containerID="646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3" exitCode=0
Oct 30 07:23:03.612002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.611975    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" event={"ID":"87305586-cd7e-4814-bd76-09c27728dafc","Type":"ContainerDied","Data":"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"}
Oct 30 07:23:03.612079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.612049    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk" event={"ID":"87305586-cd7e-4814-bd76-09c27728dafc","Type":"ContainerDied","Data":"795cf1bc17661f5766b06aeb886f8f5cf97a8c212f39a9e0cc8b6c1cc145f115"}
Oct 30 07:23:03.612079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.612072    6738 scope.go:122] "RemoveContainer" containerID="646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"
Oct 30 07:23:03.612203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.612189    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"
Oct 30 07:23:03.614391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.614357    6738 generic.go:356] "Generic (PLEG): container finished" podID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" containerID="12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce" exitCode=0
Oct 30 07:23:03.614541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.614518    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" event={"ID":"3f69fd1e-567c-468c-af03-a7f5fdf84e4a","Type":"ContainerDied","Data":"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"}
Oct 30 07:23:03.614616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.614571    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl" event={"ID":"3f69fd1e-567c-468c-af03-a7f5fdf84e4a","Type":"ContainerDied","Data":"d3d851bf2565232555234695bdd96f12d7e970731fbb0c10c3b4338fcc094d1c"}
Oct 30 07:23:03.614704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.614688    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"
Oct 30 07:23:03.624478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.624450    6738 scope.go:122] "RemoveContainer" containerID="646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"
Oct 30 07:23:03.624777 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:03.624754    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3\": container with ID starting with 646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3 not found: ID does not exist" containerID="646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"
Oct 30 07:23:03.624860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.624785    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3"} err="failed to get container status \"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3\": rpc error: code = NotFound desc = could not find container \"646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3\": container with ID starting with 646c1dc6bec9b0d5e123049f0a18e9228ba421271ad325237f00fca1486503f3 not found: ID does not exist"
Oct 30 07:23:03.624860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.624802    6738 scope.go:122] "RemoveContainer" containerID="12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"
Oct 30 07:23:03.635391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.635364    6738 scope.go:122] "RemoveContainer" containerID="12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"
Oct 30 07:23:03.635769 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:03.635740    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce\": container with ID starting with 12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce not found: ID does not exist" containerID="12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"
Oct 30 07:23:03.635846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.635783    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce"} err="failed to get container status \"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce\": rpc error: code = NotFound desc = could not find container \"12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce\": container with ID starting with 12f389d7d1985329a0c483423448f3ce07fbd5530106172569e730c0033cf2ce not found: ID does not exist"
Oct 30 07:23:03.650792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.650749    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\") pod \"87305586-cd7e-4814-bd76-09c27728dafc\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") "
Oct 30 07:23:03.650986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.650934    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\" (UniqueName: \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\") pod \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") "
Oct 30 07:23:03.651060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.650985    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\") pod \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\" (UID: \"3f69fd1e-567c-468c-af03-a7f5fdf84e4a\") "
Oct 30 07:23:03.651060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.651048    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\") pod \"87305586-cd7e-4814-bd76-09c27728dafc\" (UID: \"87305586-cd7e-4814-bd76-09c27728dafc\") "
Oct 30 07:23:03.651433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.651406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume" pod "87305586-cd7e-4814-bd76-09c27728dafc" (UID: "87305586-cd7e-4814-bd76-09c27728dafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:03.651685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.651663    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87305586-cd7e-4814-bd76-09c27728dafc-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:03.651829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.651808    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume" pod "3f69fd1e-567c-468c-af03-a7f5fdf84e4a" (UID: "3f69fd1e-567c-468c-af03-a7f5fdf84e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:03.653917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.653893    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp" pod "3f69fd1e-567c-468c-af03-a7f5fdf84e4a" (UID: "3f69fd1e-567c-468c-af03-a7f5fdf84e4a"). InnerVolumeSpecName "kube-api-access-g4gfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:03.654219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.654189    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x" pod "87305586-cd7e-4814-bd76-09c27728dafc" (UID: "87305586-cd7e-4814-bd76-09c27728dafc"). InnerVolumeSpecName "kube-api-access-td87x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:03.758344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.753085    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gfp\" (UniqueName: \"kubernetes.io/projected/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-kube-api-access-g4gfp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:03.758344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.753127    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f69fd1e-567c-468c-af03-a7f5fdf84e4a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:03.758344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.753142    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/87305586-cd7e-4814-bd76-09c27728dafc-kube-api-access-td87x\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:03.960876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.960833    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:23:03.989373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:03.967708    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2504c2-8cbf45f9f-cmfsk"]
Oct 30 07:23:04.002561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.002531    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:23:04.006601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.006566    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2504c2-84796d4f76-cxztl"]
Oct 30 07:23:04.297918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.297833    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:04.841647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.841611    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:23:04.916459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.916426    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:04.959384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.959346    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:23:04.959573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:04.959408    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:23:05.192625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.192530    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:23:05.192625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.192602    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:23:05.639310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.639258    6738 generic.go:356] "Generic (PLEG): container finished" podID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerID="cbf2078ef86c10ae5cdfcf32a2e28a4f0199e45154b1d67d497d35b3c539d6ff" exitCode=0
Oct 30 07:23:05.639808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.639464    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerDied","Data":"cbf2078ef86c10ae5cdfcf32a2e28a4f0199e45154b1d67d497d35b3c539d6ff"}
Oct 30 07:23:05.639808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.639530    6738 scope.go:122] "RemoveContainer" containerID="c5c71f4e404fbe0905b8ad1c8d9661b0b012fe4930116eff0126863ee69e2e92"
Oct 30 07:23:05.704800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.704772    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" path="/var/lib/kubelet/pods/3f69fd1e-567c-468c-af03-a7f5fdf84e4a/volumes"
Oct 30 07:23:05.705278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.705264    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87305586-cd7e-4814-bd76-09c27728dafc" path="/var/lib/kubelet/pods/87305586-cd7e-4814-bd76-09c27728dafc/volumes"
Oct 30 07:23:05.730587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.730559    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:23:05.911234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911199    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911256    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911304    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911334    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911413    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911478    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911529    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911588    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911621    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911633    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911663    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\") pod \"6d6aa838-7397-4c5d-84db-cfce027d5794\" (UID: \"6d6aa838-7397-4c5d-84db-cfce027d5794\") "
Oct 30 07:23:05.911748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911679    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:05.912116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911783    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:05.912116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.911981    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:05.912384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.912366    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:05.912462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.912386    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:05.912462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.912400    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6aa838-7397-4c5d-84db-cfce027d5794-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:05.912462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.912412    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:05.912462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.912425    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:05.913845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.913818    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:05.914069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.914048    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:05.914123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.914102    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:05.919109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:05.919083    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp" pod "6d6aa838-7397-4c5d-84db-cfce027d5794" (UID: "6d6aa838-7397-4c5d-84db-cfce027d5794"). InnerVolumeSpecName "kube-api-access-8gfvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:06.014217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.014177    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:06.014217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.014210    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:06.014217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.014220    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:06.014512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.014228    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfvp\" (UniqueName: \"kubernetes.io/projected/6d6aa838-7397-4c5d-84db-cfce027d5794-kube-api-access-8gfvp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:06.097361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.093944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:06.221313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.219246    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:06.427786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.427721    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:06.444237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.444205    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:23:06.444464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.444439    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy" containerID="cri-o://8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940" gracePeriod=30
Oct 30 07:23:06.556571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.552551    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:06.678320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.673767    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8" event={"ID":"6d6aa838-7397-4c5d-84db-cfce027d5794","Type":"ContainerDied","Data":"38767940bcd8117b9adb3c7c70a8f599a64c3ced055206aa272fdf5c052f56b9"}
Oct 30 07:23:06.678320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.673881    6738 scope.go:122] "RemoveContainer" containerID="cbf2078ef86c10ae5cdfcf32a2e28a4f0199e45154b1d67d497d35b3c539d6ff"
Oct 30 07:23:06.678320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.674017    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"
Oct 30 07:23:06.750781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.750749    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"]
Oct 30 07:23:06.751522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751501    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" containerName="payload-processing"
Oct 30 07:23:06.751522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751521    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" containerName="payload-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751535    6738 cpu_manager.go:414] "containerMap: removing container" podUID="87305586-cd7e-4814-bd76-09c27728dafc" containerName="payload-pre-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751543    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87305586-cd7e-4814-bd76-09c27728dafc" containerName="payload-pre-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751558    6738 cpu_manager.go:414] "containerMap: removing container" podUID="105f610a-3b07-4433-807a-b851ad06e842" containerName="payload-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751563    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105f610a-3b07-4433-807a-b851ad06e842" containerName="payload-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751599    6738 cpu_manager.go:414] "containerMap: removing container" podUID="63074c69-da63-414a-8123-00065b48db97" containerName="payload-pre-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751604    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63074c69-da63-414a-8123-00065b48db97" containerName="payload-pre-processing"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751616    6738 cpu_manager.go:414] "containerMap: removing container" podUID="65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" containerName="revoke-keys"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751621    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" containerName="revoke-keys"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751632    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751636    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751646    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.751671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751651    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751755    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f1d4ff-94d7-4c96-afc7-8450e6cb9eca" containerName="revoke-keys"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751765    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751771    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" containerName="istio-proxy"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751781    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69fd1e-567c-468c-af03-a7f5fdf84e4a" containerName="payload-processing"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751786    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="87305586-cd7e-4814-bd76-09c27728dafc" containerName="payload-pre-processing"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751795    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="105f610a-3b07-4433-807a-b851ad06e842" containerName="payload-processing"
Oct 30 07:23:06.752061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.751803    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="63074c69-da63-414a-8123-00065b48db97" containerName="payload-pre-processing"
Oct 30 07:23:06.755649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.755628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.764418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.764389    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:23:06.783961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.767648    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-v6dd8"]
Oct 30 07:23:06.793009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.792974    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:06.793399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.793378    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:06.802534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.802505    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"]
Oct 30 07:23:06.828421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.828310    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ns6\" (UniqueName: \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.828421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.828369    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.828665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.828428    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.929848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.929805    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ns6\" (UniqueName: \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.929848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.929847    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.930044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.929897    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.930580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.930552    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.932253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.932230    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:06.994857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:06.994819    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ns6\" (UniqueName: \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\") pod \"maas-api-revoke-sub-a0190638-v2gjn\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:07.104373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.104261    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:07.146588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.146549    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"]
Oct 30 07:23:07.150792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.150767    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.199794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.199747    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"]
Oct 30 07:23:07.235577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.235073    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4sv\" (UniqueName: \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.235577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.235305    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.235577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.235387    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.337364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.336942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.337364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.336994    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.337364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.337101    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4sv\" (UniqueName: \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.339498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.339200    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.339498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.339238    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.362723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.362644    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4sv\" (UniqueName: \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\") pod \"maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.442649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.442611    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"]
Oct 30 07:23:07.472500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.472272    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:07.532710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.532574    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:23:07.539781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.539750    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.539964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.539802    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.539964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.539842    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.539981    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540055    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540108    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540143    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540175    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540203    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\") pod \"46d51648-68a0-4881-b123-3dde837126c1\" (UID: \"46d51648-68a0-4881-b123-3dde837126c1\") "
Oct 30 07:23:07.540973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540102    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.541107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.540535    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:07.542860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.542810    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.543063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.543010    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.543409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.543374    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.546612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.546576    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.546892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.546860    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "kube-api-access-v8rpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.547418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.547254    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:07.549418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.549386    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token" pod "46d51648-68a0-4881-b123-3dde837126c1" (UID: "46d51648-68a0-4881-b123-3dde837126c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.557406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.555894    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:23:07.578991 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:07.578844    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:23:07.641749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.641712    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.641934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.641834    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.641934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.641876    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.641935    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642038    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642076    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642150    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642233    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.642322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642274    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\") pod \"73019e5b-4470-4369-9ada-d7e27ae79f54\" (UID: \"73019e5b-4470-4369-9ada-d7e27ae79f54\") "
Oct 30 07:23:07.643060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.642696    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.643060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643017    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643039    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643052    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643063    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d51648-68a0-4881-b123-3dde837126c1-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643078    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rpv\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-kube-api-access-v8rpv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643092    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643105    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643117    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d51648-68a0-4881-b123-3dde837126c1-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643129    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d51648-68a0-4881-b123-3dde837126c1-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.643452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643141    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d51648-68a0-4881-b123-3dde837126c1-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.644068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.643899    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.644253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.644218    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.644545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.644525    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:07.644903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.644868    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.645934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.645902    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:07.646086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.646068    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.648140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.648104    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.648375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.648343    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg" pod "73019e5b-4470-4369-9ada-d7e27ae79f54" (UID: "73019e5b-4470-4369-9ada-d7e27ae79f54"). InnerVolumeSpecName "kube-api-access-qjtmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.686802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.686763    6738 generic.go:356] "Generic (PLEG): container finished" podID="46d51648-68a0-4881-b123-3dde837126c1" containerID="8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b" exitCode=0
Oct 30 07:23:07.687344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.686843    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"
Oct 30 07:23:07.691715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.689514    6738 generic.go:356] "Generic (PLEG): container finished" podID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerID="be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330" exitCode=0
Oct 30 07:23:07.691715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.689709    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"
Oct 30 07:23:07.698934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.698909    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6aa838-7397-4c5d-84db-cfce027d5794" path="/var/lib/kubelet/pods/6d6aa838-7397-4c5d-84db-cfce027d5794/volumes"
Oct 30 07:23:07.699688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699663    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn" event={"ID":"06d7896a-e8eb-4303-bb51-75295a779c7f","Type":"ContainerStarted","Data":"3923a03e9bc34bbefb91022ff6c26fdfa94a3dd4ceda29912805b353e1fb5628"}
Oct 30 07:23:07.699830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699725    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" event={"ID":"46d51648-68a0-4881-b123-3dde837126c1","Type":"ContainerDied","Data":"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"}
Oct 30 07:23:07.699830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699767    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz" event={"ID":"46d51648-68a0-4881-b123-3dde837126c1","Type":"ContainerDied","Data":"02745c1fd135a3de50a735e567a86c1a8e0d96f25a2bdde3a88a0cd4444ad1de"}
Oct 30 07:23:07.699830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699782    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerDied","Data":"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"}
Oct 30 07:23:07.699830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699828    6738 scope.go:122] "RemoveContainer" containerID="8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"
Oct 30 07:23:07.700035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.699936    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz" event={"ID":"73019e5b-4470-4369-9ada-d7e27ae79f54","Type":"ContainerDied","Data":"90cc7b7808ef1447cb11f84abf5c9aabac9e7ec09c8d394611721530d53ff735"}
Oct 30 07:23:07.712185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.712003    6738 scope.go:122] "RemoveContainer" containerID="8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"
Oct 30 07:23:07.712418 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:07.712393    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b\": container with ID starting with 8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b not found: ID does not exist" containerID="8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"
Oct 30 07:23:07.712471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.712431    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b"} err="failed to get container status \"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b\": rpc error: code = NotFound desc = could not find container \"8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b\": container with ID starting with 8882bb9d7e113ae115b6042df72dcee2d5206fb68b9f1b214ae5a587c425e50b not found: ID does not exist"
Oct 30 07:23:07.712471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.712455    6738 scope.go:122] "RemoveContainer" containerID="be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"
Oct 30 07:23:07.725755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.725729    6738 scope.go:122] "RemoveContainer" containerID="577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"
Oct 30 07:23:07.736431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.736407    6738 scope.go:122] "RemoveContainer" containerID="be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"
Oct 30 07:23:07.736904 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:07.736876    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330\": container with ID starting with be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330 not found: ID does not exist" containerID="be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"
Oct 30 07:23:07.737030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.736916    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330"} err="failed to get container status \"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330\": rpc error: code = NotFound desc = could not find container \"be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330\": container with ID starting with be8b233591a28f36dae4561705b91abcbff315fee8c01d39e23e4625f9f90330 not found: ID does not exist"
Oct 30 07:23:07.737030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.736941    6738 scope.go:122] "RemoveContainer" containerID="577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"
Oct 30 07:23:07.737327 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:07.737282    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf\": container with ID starting with 577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf not found: ID does not exist" containerID="577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"
Oct 30 07:23:07.737416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.737342    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf"} err="failed to get container status \"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf\": rpc error: code = NotFound desc = could not find container \"577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf\": container with ID starting with 577c18f0b86dc2599e1923f39be49a205d8b8156c9ccb3289231f544ed7e6adf not found: ID does not exist"
Oct 30 07:23:07.744161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744136    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744161    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744170    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744178    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744188    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73019e5b-4470-4369-9ada-d7e27ae79f54-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744196    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744205    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjtmg\" (UniqueName: \"kubernetes.io/projected/73019e5b-4470-4369-9ada-d7e27ae79f54-kube-api-access-qjtmg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.744362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.744213    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73019e5b-4470-4369-9ada-d7e27ae79f54-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.758892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.758860    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:23:07.764044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.764014    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-bgzbz"]
Oct 30 07:23:07.798509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.798470    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:23:07.802397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.802369    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-rq4kz"]
Oct 30 07:23:07.838583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:07.838539    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"]
Oct 30 07:23:07.840989 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:07.840956    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28d5b5d_9812_419d_ae8e_000660d917ad.slice/crio-986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba WatchSource:0}: Error finding container 986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba: Status 404 returned error can't find the container with id 986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba
Oct 30 07:23:08.396946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.396909    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:23:08.435398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.435356    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:23:08.535953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.535794    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:08.673623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.669760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:08.728002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.727954    6738 generic.go:356] "Generic (PLEG): container finished" podID="06d7896a-e8eb-4303-bb51-75295a779c7f" containerID="74ee9ba7cb4b39605e18fa7d85c9916deb7872115b4496c219ed598325f4d71c" exitCode=0
Oct 30 07:23:08.728512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.728061    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn" event={"ID":"06d7896a-e8eb-4303-bb51-75295a779c7f","Type":"ContainerDied","Data":"74ee9ba7cb4b39605e18fa7d85c9916deb7872115b4496c219ed598325f4d71c"}
Oct 30 07:23:08.731501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.731467    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h" event={"ID":"e28d5b5d-9812-419d-ae8e-000660d917ad","Type":"ContainerStarted","Data":"54dd60bfbb7f829e395bf60c4e82ea127c357d206e4d6bef75fc19d7037d20b5"}
Oct 30 07:23:08.731657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.731527    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h" event={"ID":"e28d5b5d-9812-419d-ae8e-000660d917ad","Type":"ContainerStarted","Data":"986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba"}
Oct 30 07:23:08.811284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.811174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:08.837636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:08.837551    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h" podStartSLOduration=1.8375178779999999 podStartE2EDuration="1.837517878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:08.836450543 +0000 UTC m=+1549.269214435" watchObservedRunningTime="2026-10-30 07:23:08.837517878 +0000 UTC m=+1549.270281772"
Oct 30 07:23:09.693807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.693772    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d51648-68a0-4881-b123-3dde837126c1" path="/var/lib/kubelet/pods/46d51648-68a0-4881-b123-3dde837126c1/volumes"
Oct 30 07:23:09.694276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.694261    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" path="/var/lib/kubelet/pods/73019e5b-4470-4369-9ada-d7e27ae79f54/volumes"
Oct 30 07:23:09.745130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.745075    6738 generic.go:356] "Generic (PLEG): container finished" podID="e28d5b5d-9812-419d-ae8e-000660d917ad" containerID="54dd60bfbb7f829e395bf60c4e82ea127c357d206e4d6bef75fc19d7037d20b5" exitCode=0
Oct 30 07:23:09.745919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.745167    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h" event={"ID":"e28d5b5d-9812-419d-ae8e-000660d917ad","Type":"ContainerDied","Data":"54dd60bfbb7f829e395bf60c4e82ea127c357d206e4d6bef75fc19d7037d20b5"}
Oct 30 07:23:09.928202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.928175    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:09.992501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.992417    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\") pod \"06d7896a-e8eb-4303-bb51-75295a779c7f\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") "
Oct 30 07:23:09.992501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.992464    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\" (UniqueName: \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\") pod \"06d7896a-e8eb-4303-bb51-75295a779c7f\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") "
Oct 30 07:23:09.992501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.992499    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\") pod \"06d7896a-e8eb-4303-bb51-75295a779c7f\" (UID: \"06d7896a-e8eb-4303-bb51-75295a779c7f\") "
Oct 30 07:23:09.993229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.993200    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca" pod "06d7896a-e8eb-4303-bb51-75295a779c7f" (UID: "06d7896a-e8eb-4303-bb51-75295a779c7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:09.995098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.995056    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6" pod "06d7896a-e8eb-4303-bb51-75295a779c7f" (UID: "06d7896a-e8eb-4303-bb51-75295a779c7f"). InnerVolumeSpecName "kube-api-access-d5ns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:09.995468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:09.995440    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca" pod "06d7896a-e8eb-4303-bb51-75295a779c7f" (UID: "06d7896a-e8eb-4303-bb51-75295a779c7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:10.093958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.093919    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:10.093958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.093949    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ns6\" (UniqueName: \"kubernetes.io/projected/06d7896a-e8eb-4303-bb51-75295a779c7f-kube-api-access-d5ns6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:10.093958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.093960    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d7896a-e8eb-4303-bb51-75295a779c7f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:10.440173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.440133    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:10.755517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.755433    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"
Oct 30 07:23:10.755517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.755452    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn" event={"ID":"06d7896a-e8eb-4303-bb51-75295a779c7f","Type":"ContainerDied","Data":"3923a03e9bc34bbefb91022ff6c26fdfa94a3dd4ceda29912805b353e1fb5628"}
Oct 30 07:23:10.756047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.755520    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3923a03e9bc34bbefb91022ff6c26fdfa94a3dd4ceda29912805b353e1fb5628"
Oct 30 07:23:10.928285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:10.928256    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:11.027873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.027791    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\") pod \"e28d5b5d-9812-419d-ae8e-000660d917ad\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") "
Oct 30 07:23:11.028037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.027937    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\" (UniqueName: \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\") pod \"e28d5b5d-9812-419d-ae8e-000660d917ad\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") "
Oct 30 07:23:11.028037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.027973    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\") pod \"e28d5b5d-9812-419d-ae8e-000660d917ad\" (UID: \"e28d5b5d-9812-419d-ae8e-000660d917ad\") "
Oct 30 07:23:11.028196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.028171    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca" pod "e28d5b5d-9812-419d-ae8e-000660d917ad" (UID: "e28d5b5d-9812-419d-ae8e-000660d917ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:11.028649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.028627    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.030180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.030156    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca" pod "e28d5b5d-9812-419d-ae8e-000660d917ad" (UID: "e28d5b5d-9812-419d-ae8e-000660d917ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:11.030284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.030202    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv" pod "e28d5b5d-9812-419d-ae8e-000660d917ad" (UID: "e28d5b5d-9812-419d-ae8e-000660d917ad"). InnerVolumeSpecName "kube-api-access-jk4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:11.129820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.129785    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28d5b5d-9812-419d-ae8e-000660d917ad-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.129820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.129816    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4sv\" (UniqueName: \"kubernetes.io/projected/e28d5b5d-9812-419d-ae8e-000660d917ad-kube-api-access-jk4sv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.726001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.725975    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:23:11.766027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.765995    6738 generic.go:356] "Generic (PLEG): container finished" podID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerID="8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940" exitCode=0
Oct 30 07:23:11.766553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.766069    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"
Oct 30 07:23:11.766553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.766076    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerDied","Data":"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"}
Oct 30 07:23:11.766553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.766171    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2" event={"ID":"cb88b431-45a8-43cd-b43f-5feb2b706559","Type":"ContainerDied","Data":"830dc10a60bc17db9f29d6d4ad5fe8b42c224b9142ac731f82bd601053efac78"}
Oct 30 07:23:11.766553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.766188    6738 scope.go:122] "RemoveContainer" containerID="8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"
Oct 30 07:23:11.767678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.767573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h" event={"ID":"e28d5b5d-9812-419d-ae8e-000660d917ad","Type":"ContainerDied","Data":"986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba"}
Oct 30 07:23:11.767678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.767605    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986c8cc05ffa01604b95f6584116dfddd1029605d857859f4049ea412ee316ba"
Oct 30 07:23:11.767678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.767661    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"
Oct 30 07:23:11.770238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.770206    6738 generic.go:356] "Generic (PLEG): container finished" podID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerID="4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25" exitCode=0
Oct 30 07:23:11.770373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.770253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerDied","Data":"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"}
Oct 30 07:23:11.770708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.770694    6738 scope.go:122] "RemoveContainer" containerID="4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"
Oct 30 07:23:11.772234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.772196    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:11.772340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.772287    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:11.772389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.772376    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:11.778202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.778172    6738 scope.go:122] "RemoveContainer" containerID="9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"
Oct 30 07:23:11.801602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.801570    6738 scope.go:122] "RemoveContainer" containerID="8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"
Oct 30 07:23:11.802357 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:11.802327    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940\": container with ID starting with 8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940 not found: ID does not exist" containerID="8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"
Oct 30 07:23:11.802690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.802653    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940"} err="failed to get container status \"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940\": rpc error: code = NotFound desc = could not find container \"8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940\": container with ID starting with 8f24c3a3189bdcc90586ac33f35936ea8b4cedb09fab490e11799f8662fba940 not found: ID does not exist"
Oct 30 07:23:11.802690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.802691    6738 scope.go:122] "RemoveContainer" containerID="9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"
Oct 30 07:23:11.803090 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:11.803065    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5\": container with ID starting with 9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5 not found: ID does not exist" containerID="9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"
Oct 30 07:23:11.803161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.803098    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5"} err="failed to get container status \"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5\": rpc error: code = NotFound desc = could not find container \"9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5\": container with ID starting with 9cab1e18d4fb85094054dc0f447a0cde7decf7e174a56f43a9317225706355f5 not found: ID does not exist"
Oct 30 07:23:11.842483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842458    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842557    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842575    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842593    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842620    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842679    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.842918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842749    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:11.842918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842818    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.843094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842937    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.843094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.842979    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\") pod \"cb88b431-45a8-43cd-b43f-5feb2b706559\" (UID: \"cb88b431-45a8-43cd-b43f-5feb2b706559\") "
Oct 30 07:23:11.843094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843077    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:11.843252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843086    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:11.843252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843041    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:11.843521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843496    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:11.843739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843712    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.843881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843741    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb88b431-45a8-43cd-b43f-5feb2b706559-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.843881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843757    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.843881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843773    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.843881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.843787    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.845097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.845070    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:11.845222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.845075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "kube-api-access-gqz9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:11.845551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.845531    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:11.845551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.845542    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token" pod "cb88b431-45a8-43cd-b43f-5feb2b706559" (UID: "cb88b431-45a8-43cd-b43f-5feb2b706559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:11.945700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.945653    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.945700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.945685    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.945700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.945698    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb88b431-45a8-43cd-b43f-5feb2b706559-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:11.945937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:11.945711    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqz9g\" (UniqueName: \"kubernetes.io/projected/cb88b431-45a8-43cd-b43f-5feb2b706559-kube-api-access-gqz9g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:12.114586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.114548    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:23:12.118384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.118350    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2504c2-openshift-default-5f54674b8f-tj5p2"]
Oct 30 07:23:12.394216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.394157    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:12.800823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.800719    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerStarted","Data":"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"}
Oct 30 07:23:12.860734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.860700    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:23:12.861139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.861114    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:12.861222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:12.861172    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:13.397663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.397619    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" podUID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" containerName="payload-processing" containerID="cri-o://2ecbaca7c2d69fd04ff9f83d6051aaefcf37ec9bfdc107785a2a23d0412afcfc" gracePeriod=25
Oct 30 07:23:13.435938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.435892    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" podUID="799f1cf6-674f-4f2c-a873-fcc11bff17da" containerName="payload-pre-processing" containerID="cri-o://db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844" gracePeriod=25
Oct 30 07:23:13.712370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.712214    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" path="/var/lib/kubelet/pods/cb88b431-45a8-43cd-b43f-5feb2b706559/volumes"
Oct 30 07:23:13.816805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.816772    6738 generic.go:356] "Generic (PLEG): container finished" podID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" containerID="2ecbaca7c2d69fd04ff9f83d6051aaefcf37ec9bfdc107785a2a23d0412afcfc" exitCode=0
Oct 30 07:23:13.817172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.816914    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" event={"ID":"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab","Type":"ContainerDied","Data":"2ecbaca7c2d69fd04ff9f83d6051aaefcf37ec9bfdc107785a2a23d0412afcfc"}
Oct 30 07:23:13.817172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.816952    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s" event={"ID":"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab","Type":"ContainerDied","Data":"9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920"}
Oct 30 07:23:13.817172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.816968    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8a4672c955f7f51ad76a3d271ef4be9df3f49931cbc52d43cc12fcce8e3920"
Oct 30 07:23:13.818398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.818379    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:23:13.819334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.819284    6738 generic.go:356] "Generic (PLEG): container finished" podID="799f1cf6-674f-4f2c-a873-fcc11bff17da" containerID="db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844" exitCode=0
Oct 30 07:23:13.819334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.819323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" event={"ID":"799f1cf6-674f-4f2c-a873-fcc11bff17da","Type":"ContainerDied","Data":"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"}
Oct 30 07:23:13.819483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.819375    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m" event={"ID":"799f1cf6-674f-4f2c-a873-fcc11bff17da","Type":"ContainerDied","Data":"1fd9af647b989e590a2c51fbf90f44cc06812bd57469ee624f547531af93ae27"}
Oct 30 07:23:13.819483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.819398    6738 scope.go:122] "RemoveContainer" containerID="db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"
Oct 30 07:23:13.821586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.821571    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:23:13.830351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.830329    6738 scope.go:122] "RemoveContainer" containerID="db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"
Oct 30 07:23:13.830734 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:13.830708    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844\": container with ID starting with db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844 not found: ID does not exist" containerID="db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"
Oct 30 07:23:13.830834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.830743    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844"} err="failed to get container status \"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844\": rpc error: code = NotFound desc = could not find container \"db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844\": container with ID starting with db18586170051405f1bbba6a5ea6604a5671b35a9105b50e39649b85c15d4844 not found: ID does not exist"
Oct 30 07:23:13.861921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.861862    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:13.863688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:13.863646    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:14.009816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.009718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\" (UniqueName: \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\") pod \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") "
Oct 30 07:23:14.010010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.009983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\" (UniqueName: \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\") pod \"799f1cf6-674f-4f2c-a873-fcc11bff17da\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") "
Oct 30 07:23:14.010080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010062    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\") pod \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\" (UID: \"e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab\") "
Oct 30 07:23:14.010155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010140    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\") pod \"799f1cf6-674f-4f2c-a873-fcc11bff17da\" (UID: \"799f1cf6-674f-4f2c-a873-fcc11bff17da\") "
Oct 30 07:23:14.010807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010536    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume" pod "e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" (UID: "e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:14.010807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010577    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume" pod "799f1cf6-674f-4f2c-a873-fcc11bff17da" (UID: "799f1cf6-674f-4f2c-a873-fcc11bff17da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:14.011031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010974    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:14.011031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.010995    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799f1cf6-674f-4f2c-a873-fcc11bff17da-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:14.011990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.011971    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2" pod "e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" (UID: "e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab"). InnerVolumeSpecName "kube-api-access-9msw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:14.012463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.012440    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl" pod "799f1cf6-674f-4f2c-a873-fcc11bff17da" (UID: "799f1cf6-674f-4f2c-a873-fcc11bff17da"). InnerVolumeSpecName "kube-api-access-zksxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:14.112569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.112534    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msw2\" (UniqueName: \"kubernetes.io/projected/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab-kube-api-access-9msw2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:14.112569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.112566    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zksxl\" (UniqueName: \"kubernetes.io/projected/799f1cf6-674f-4f2c-a873-fcc11bff17da-kube-api-access-zksxl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:14.828179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.828098    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"
Oct 30 07:23:14.828611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.828253    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"
Oct 30 07:23:14.860764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.860732    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:14.860981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.860790    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:14.909832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.909803    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:23:14.912509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.912485    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-7657884bc9-f4r2m"]
Oct 30 07:23:14.949685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.949648    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:23:14.958055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:14.958023    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-7f4b8988d6-k9f8s"]
Oct 30 07:23:15.092438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:15.092336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:15.699540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:15.699500    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799f1cf6-674f-4f2c-a873-fcc11bff17da" path="/var/lib/kubelet/pods/799f1cf6-674f-4f2c-a873-fcc11bff17da/volumes"
Oct 30 07:23:15.700072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:15.700049    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" path="/var/lib/kubelet/pods/e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab/volumes"
Oct 30 07:23:15.860640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:15.860605    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:15.861059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:15.860664    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:16.863329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:16.861770    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:16.863329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:16.861828    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:17.149793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.149747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:17.421920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.421831    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:23:17.454012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.453980    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:23:17.567181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.567148    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:17.710375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.705128    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:17.860709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.860668    6738 patch_prober.go:29] interesting pod/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:23:17.860927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:17.860740    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 07:23:18.865977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:18.865949    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:23:18.866468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:18.866225    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:23:18.867116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:18.867100    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:23:19.464053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:19.464016    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:19.864632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:19.864545    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerName="main" containerID="cri-o://5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8" gracePeriod=45
Oct 30 07:23:19.897351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:19.897267    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.187:8000/ready\": dial tcp 10.133.0.187:8000: connect: connection refused"
Oct 30 07:23:19.992464 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:19.967958    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b68e9da_e0cf_4a6c_a468_86a861ec92e1.slice/crio-conmon-5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 07:23:20.201272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.201243    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:23:20.223856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.223827    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.223860    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.223898    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.223940    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224041    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224145    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\" (UniqueName: \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\") pod \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\" (UID: \"0b68e9da-e0cf-4a6c-a468-86a861ec92e1\") "
Oct 30 07:23:20.224236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224139    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:20.224236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224185    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:20.224236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224214    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:20.224744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224725    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.224825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224748    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.224825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.224763    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.226419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.226386    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:20.226419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.226386    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:20.226578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.226474    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k" pod "0b68e9da-e0cf-4a6c-a468-86a861ec92e1" (UID: "0b68e9da-e0cf-4a6c-a468-86a861ec92e1"). InnerVolumeSpecName "kube-api-access-wjt7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:20.326374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.326339    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.326374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.326368    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.326374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.326378    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjt7k\" (UniqueName: \"kubernetes.io/projected/0b68e9da-e0cf-4a6c-a468-86a861ec92e1-kube-api-access-wjt7k\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.756623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.756584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:20.927077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.927038    6738 generic.go:356] "Generic (PLEG): container finished" podID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerID="5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8" exitCode=0
Oct 30 07:23:20.927649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.927091    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"
Oct 30 07:23:20.927649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.927111    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" event={"ID":"0b68e9da-e0cf-4a6c-a468-86a861ec92e1","Type":"ContainerDied","Data":"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"}
Oct 30 07:23:20.927649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.927156    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj" event={"ID":"0b68e9da-e0cf-4a6c-a468-86a861ec92e1","Type":"ContainerDied","Data":"9922f61e4ee233f50b96f89d0ca6e38c48f6b22745e0641b74f1ab0546f7214d"}
Oct 30 07:23:20.927649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.927177    6738 scope.go:122] "RemoveContainer" containerID="5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"
Oct 30 07:23:20.938311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.938280    6738 scope.go:122] "RemoveContainer" containerID="5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"
Oct 30 07:23:20.938665 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:20.938643    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8\": container with ID starting with 5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8 not found: ID does not exist" containerID="5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"
Oct 30 07:23:20.938737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.938677    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8"} err="failed to get container status \"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8\": rpc error: code = NotFound desc = could not find container \"5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8\": container with ID starting with 5c8da1c58a50a9414680c38de830af47950faadecfe28d242ab384e8c431fcf8 not found: ID does not exist"
Oct 30 07:23:20.993926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:20.993532    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:23:21.003757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:21.003723    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a51baa/praxis-route-a51baa-kserve-65c7c4f8f5-jgkqj"]
Oct 30 07:23:21.393457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:21.393408    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:21.698960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:21.698882    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" path="/var/lib/kubelet/pods/0b68e9da-e0cf-4a6c-a468-86a861ec92e1/volumes"
Oct 30 07:23:22.298092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298052    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"]
Oct 30 07:23:22.298872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298848    6738 cpu_manager.go:414] "containerMap: removing container" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298873    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298901    6738 cpu_manager.go:414] "containerMap: removing container" podUID="799f1cf6-674f-4f2c-a873-fcc11bff17da" containerName="payload-pre-processing"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298909    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799f1cf6-674f-4f2c-a873-fcc11bff17da" containerName="payload-pre-processing"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298930    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298937    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298949    6738 cpu_manager.go:414] "containerMap: removing container" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298956    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298975    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerName="main"
Oct 30 07:23:22.298985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.298983    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerName="main"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299000    6738 cpu_manager.go:414] "containerMap: removing container" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299007    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299019    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e28d5b5d-9812-419d-ae8e-000660d917ad" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299027    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28d5b5d-9812-419d-ae8e-000660d917ad" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299045    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299053    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299076    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" containerName="payload-processing"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299084    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" containerName="payload-processing"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299115    6738 cpu_manager.go:414] "containerMap: removing container" podUID="06d7896a-e8eb-4303-bb51-75295a779c7f" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299123    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d7896a-e8eb-4303-bb51-75295a779c7f" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299259    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d7896a-e8eb-4303-bb51-75295a779c7f" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299272    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299285    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="799f1cf6-674f-4f2c-a873-fcc11bff17da" containerName="payload-pre-processing"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299316    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299327    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b68e9da-e0cf-4a6c-a468-86a861ec92e1" containerName="main"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299337    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d51648-68a0-4881-b123-3dde837126c1" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299344    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28d5b5d-9812-419d-ae8e-000660d917ad" containerName="revoke-keys"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299353    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb88b431-45a8-43cd-b43f-5feb2b706559" containerName="istio-proxy"
Oct 30 07:23:22.299478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.299365    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b4bd41-42ee-48cb-9ef4-d1899e3d15ab" containerName="payload-processing"
Oct 30 07:23:22.302890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.302868    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.320217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.320184    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"]
Oct 30 07:23:22.329127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.329100    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:22.329353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.329191    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:22.362185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.362149    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwjt\" (UniqueName: \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.362375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.362198    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.362438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.362415    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.423136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.423091    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" podUID="c8fceb53-e30c-4233-b8d7-33996640a59d" containerName="payload-processing" containerID="cri-o://e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e" gracePeriod=25
Oct 30 07:23:22.454802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.454732    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" podUID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" containerName="payload-pre-processing" containerID="cri-o://bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406" gracePeriod=25
Oct 30 07:23:22.463896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.463860    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.464045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.464028    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwjt\" (UniqueName: \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.464096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.464073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.464674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.464650    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.466239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.466221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.488069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.488035    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwjt\" (UniqueName: \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\") pod \"maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.616377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.616331    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:22.767578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.767546    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:23:22.807752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.807662    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:23:22.870148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.870062    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\" (UniqueName: \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\") pod \"c8fceb53-e30c-4233-b8d7-33996640a59d\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") "
Oct 30 07:23:22.870362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.870151    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\" (UniqueName: \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\") pod \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") "
Oct 30 07:23:22.870466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.870398    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\") pod \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\" (UID: \"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652\") "
Oct 30 07:23:22.870466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.870444    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\") pod \"c8fceb53-e30c-4233-b8d7-33996640a59d\" (UID: \"c8fceb53-e30c-4233-b8d7-33996640a59d\") "
Oct 30 07:23:22.870807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.870784    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume" pod "b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" (UID: "b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:22.871128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.871108    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:22.871230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.871156    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume" pod "c8fceb53-e30c-4233-b8d7-33996640a59d" (UID: "c8fceb53-e30c-4233-b8d7-33996640a59d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:22.872982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.872957    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr" pod "b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" (UID: "b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652"). InnerVolumeSpecName "kube-api-access-wncwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:22.873095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.872993    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48" pod "c8fceb53-e30c-4233-b8d7-33996640a59d" (UID: "c8fceb53-e30c-4233-b8d7-33996640a59d"). InnerVolumeSpecName "kube-api-access-r5d48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:22.958658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.958616    6738 generic.go:356] "Generic (PLEG): container finished" podID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" containerID="bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406" exitCode=0
Oct 30 07:23:22.958962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.958743    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"
Oct 30 07:23:22.959101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.958994    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" event={"ID":"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652","Type":"ContainerDied","Data":"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"}
Oct 30 07:23:22.959437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.959416    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk" event={"ID":"b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652","Type":"ContainerDied","Data":"0d98da672ed060504b5563b2f418c1351ed40cd7b7960ccccdbe6a355572cebf"}
Oct 30 07:23:22.959566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.959555    6738 scope.go:122] "RemoveContainer" containerID="bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"
Oct 30 07:23:22.962109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.962083    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"]
Oct 30 07:23:22.965398 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:22.965372    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5aff3db_424d_4e18_bcef_a27244fb3f6c.slice/crio-6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e WatchSource:0}: Error finding container 6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e: Status 404 returned error can't find the container with id 6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e
Oct 30 07:23:22.970792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.970766    6738 generic.go:356] "Generic (PLEG): container finished" podID="c8fceb53-e30c-4233-b8d7-33996640a59d" containerID="e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e" exitCode=0
Oct 30 07:23:22.970901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.970889    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" event={"ID":"c8fceb53-e30c-4233-b8d7-33996640a59d","Type":"ContainerDied","Data":"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"}
Oct 30 07:23:22.970964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.970932    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp" event={"ID":"c8fceb53-e30c-4233-b8d7-33996640a59d","Type":"ContainerDied","Data":"65507570715edc5266cb3bad55b87bf1682bbd7cae76afc883f9358c9def242c"}
Oct 30 07:23:22.970964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.970932    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"
Oct 30 07:23:22.972621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.972588    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8fceb53-e30c-4233-b8d7-33996640a59d-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:22.972735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.972635    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d48\" (UniqueName: \"kubernetes.io/projected/c8fceb53-e30c-4233-b8d7-33996640a59d-kube-api-access-r5d48\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:22.972735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.972651    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wncwr\" (UniqueName: \"kubernetes.io/projected/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652-kube-api-access-wncwr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:22.990181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.990158    6738 scope.go:122] "RemoveContainer" containerID="bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"
Oct 30 07:23:22.990581 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:22.990550    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406\": container with ID starting with bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406 not found: ID does not exist" containerID="bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"
Oct 30 07:23:22.990703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.990591    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406"} err="failed to get container status \"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406\": rpc error: code = NotFound desc = could not find container \"bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406\": container with ID starting with bd17051a12d25acb709801150f60019e81d25a58cf6c92aed7a07bff3c6fa406 not found: ID does not exist"
Oct 30 07:23:22.990703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:22.990615    6738 scope.go:122] "RemoveContainer" containerID="e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"
Oct 30 07:23:23.002682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.002659    6738 scope.go:122] "RemoveContainer" containerID="e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"
Oct 30 07:23:23.003065 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:23.003029    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e\": container with ID starting with e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e not found: ID does not exist" containerID="e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"
Oct 30 07:23:23.003177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.003078    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e"} err="failed to get container status \"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e\": rpc error: code = NotFound desc = could not find container \"e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e\": container with ID starting with e0cd19499e842e40a4a0a311fa395ac6109fb1f7c8a6f61fe78eaf3e16b1127e not found: ID does not exist"
Oct 30 07:23:23.013859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.013797    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:23:23.021079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.021048    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-411c9d-67bc49bcf9-ptnbk"]
Oct 30 07:23:23.075709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.075554    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:23:23.081729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.081687    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-411c9d-7c6ffffbb-rs8rp"]
Oct 30 07:23:23.425021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.424978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:23.692570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.692531    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" path="/var/lib/kubelet/pods/b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652/volumes"
Oct 30 07:23:23.692984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.692969    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fceb53-e30c-4233-b8d7-33996640a59d" path="/var/lib/kubelet/pods/c8fceb53-e30c-4233-b8d7-33996640a59d/volumes"
Oct 30 07:23:23.981895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.981778    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c" event={"ID":"e5aff3db-424d-4e18-bcef-a27244fb3f6c","Type":"ContainerStarted","Data":"40fefe15a955aaf22c4fee287fd8e036efbca4b50742636f7b293f308be6f3d7"}
Oct 30 07:23:23.982195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:23.982163    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c" event={"ID":"e5aff3db-424d-4e18-bcef-a27244fb3f6c","Type":"ContainerStarted","Data":"6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e"}
Oct 30 07:23:24.025236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.025178    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c" podStartSLOduration=2.025163445 podStartE2EDuration="2.025163445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:24.022948096 +0000 UTC m=+1564.455712001" watchObservedRunningTime="2026-10-30 07:23:24.025163445 +0000 UTC m=+1564.457927338"
Oct 30 07:23:24.405255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.405222    6738 scope.go:122] "RemoveContainer" containerID="55328ea7571eae6cc0e398eaeece63d4993bfddff8e566f8721d92761a02e34d"
Oct 30 07:23:24.415898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.415857    6738 scope.go:122] "RemoveContainer" containerID="1010b908de74b53da101dc3e183fedde2257123b9f00e80263b77f0acb372207"
Oct 30 07:23:24.426253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.426226    6738 scope.go:122] "RemoveContainer" containerID="ed0eb6445a3ea5284828a1a128374c9795fe4a02fb3c396d502521af1da31eb2"
Oct 30 07:23:24.993524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.993482    6738 generic.go:356] "Generic (PLEG): container finished" podID="e5aff3db-424d-4e18-bcef-a27244fb3f6c" containerID="40fefe15a955aaf22c4fee287fd8e036efbca4b50742636f7b293f308be6f3d7" exitCode=0
Oct 30 07:23:24.993685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:24.993564    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c" event={"ID":"e5aff3db-424d-4e18-bcef-a27244fb3f6c","Type":"ContainerDied","Data":"40fefe15a955aaf22c4fee287fd8e036efbca4b50742636f7b293f308be6f3d7"}
Oct 30 07:23:25.445961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:25.445921    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:26.175223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.175193    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:26.221602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.221567    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\") pod \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") "
Oct 30 07:23:26.221796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.221622    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\" (UniqueName: \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\") pod \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") "
Oct 30 07:23:26.221880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.221822    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\") pod \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\" (UID: \"e5aff3db-424d-4e18-bcef-a27244fb3f6c\") "
Oct 30 07:23:26.222015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.221988    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca" pod "e5aff3db-424d-4e18-bcef-a27244fb3f6c" (UID: "e5aff3db-424d-4e18-bcef-a27244fb3f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:26.222453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.222433    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:26.224000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.223967    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca" pod "e5aff3db-424d-4e18-bcef-a27244fb3f6c" (UID: "e5aff3db-424d-4e18-bcef-a27244fb3f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:26.224250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.224227    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt" pod "e5aff3db-424d-4e18-bcef-a27244fb3f6c" (UID: "e5aff3db-424d-4e18-bcef-a27244fb3f6c"). InnerVolumeSpecName "kube-api-access-brwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:26.323372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.323267    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwjt\" (UniqueName: \"kubernetes.io/projected/e5aff3db-424d-4e18-bcef-a27244fb3f6c-kube-api-access-brwjt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:26.323372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:26.323309    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5aff3db-424d-4e18-bcef-a27244fb3f6c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:27.018320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.018268    6738 generic.go:356] "Generic (PLEG): container finished" podID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerID="4fa4f12c860e46186b0d8c037b25b425513e020758b674d81c424e948cd2f7ac" exitCode=0
Oct 30 07:23:27.018869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.018449    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerDied","Data":"4fa4f12c860e46186b0d8c037b25b425513e020758b674d81c424e948cd2f7ac"}
Oct 30 07:23:27.019029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.019008    6738 scope.go:122] "RemoveContainer" containerID="4fa4f12c860e46186b0d8c037b25b425513e020758b674d81c424e948cd2f7ac"
Oct 30 07:23:27.020363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.020329    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c" event={"ID":"e5aff3db-424d-4e18-bcef-a27244fb3f6c","Type":"ContainerDied","Data":"6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e"}
Oct 30 07:23:27.020462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.020414    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a7e0bc3b613677612c7b5e0fcff97bdb24b404bc78569543c7521281f89359e"
Oct 30 07:23:27.020462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.020339    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"
Oct 30 07:23:27.021112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.021079    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:27.021190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.021170    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:27.021244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.021224    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:27.387127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:27.387092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:28.031216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:28.031181    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerStarted","Data":"5c91fe7b706116968dba21268be4f4662c83544381bb80a865aa0a8b283f2821"}
Oct 30 07:23:28.439521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:28.439482    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:23:28.441007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:28.440972    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:28.441144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:28.441025    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:28.793871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:28.793770    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:29.025429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.025380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.239951    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240851    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e5aff3db-424d-4e18-bcef-a27244fb3f6c" containerName="revoke-keys"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240871    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aff3db-424d-4e18-bcef-a27244fb3f6c" containerName="revoke-keys"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240898    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c8fceb53-e30c-4233-b8d7-33996640a59d" containerName="payload-processing"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240906    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fceb53-e30c-4233-b8d7-33996640a59d" containerName="payload-processing"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240924    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" containerName="payload-pre-processing"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.240931    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" containerName="payload-pre-processing"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.241134    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="73019e5b-4470-4369-9ada-d7e27ae79f54" containerName="istio-proxy"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.241150    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aff3db-424d-4e18-bcef-a27244fb3f6c" containerName="revoke-keys"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.241163    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fceb53-e30c-4233-b8d7-33996640a59d" containerName="payload-processing"
Oct 30 07:23:29.244318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.241173    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d7eb3b-a68d-46e3-8ae4-a8cb3871b652" containerName="payload-pre-processing"
Oct 30 07:23:29.276356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.261193    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.277335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.272368    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:23:29.312318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.295366    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-89664e\""
Oct 30 07:23:29.368327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.340065    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.368327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.340178    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj48l\" (UniqueName: \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.391331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.372874    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:23:29.407693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.407660    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.409872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.409839    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:23:29.423913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.423878    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-89664e-dockercfg-dk6zz\""
Oct 30 07:23:29.439881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.439843    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:29.440253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.440212    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:29.441319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.441091    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj48l\" (UniqueName: \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.441319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.441173    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.441719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.441627    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whpk\" (UniqueName: \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.441719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.441682    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.443826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.442797    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.484319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.483078    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj48l\" (UniqueName: \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\") pod \"payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.545253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.545161    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.545454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.545304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whpk\" (UniqueName: \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.545925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.545905    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.581861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.578106    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whpk\" (UniqueName: \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\") pod \"payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:29.655318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.651405    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:29.741115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:29.741079    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:30.297073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.296942    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:23:30.299849 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:30.299817    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10baaa70_ea0c_482a_9e67_968a8573954e.slice/crio-787d00fd2a1ca82e9249a2e51091a71929db0c9721b015e5bda16b1a8ac3451b WatchSource:0}: Error finding container 787d00fd2a1ca82e9249a2e51091a71929db0c9721b015e5bda16b1a8ac3451b: Status 404 returned error can't find the container with id 787d00fd2a1ca82e9249a2e51091a71929db0c9721b015e5bda16b1a8ac3451b
Oct 30 07:23:30.300716 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:30.300678    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b2b062_3098_48b5_b850_560c8fdaac08.slice/crio-7eb5c5c93ad5ecdb37566a9b0282dc60d5ee56b58ea17bc197c51b32a055ce9c WatchSource:0}: Error finding container 7eb5c5c93ad5ecdb37566a9b0282dc60d5ee56b58ea17bc197c51b32a055ce9c: Status 404 returned error can't find the container with id 7eb5c5c93ad5ecdb37566a9b0282dc60d5ee56b58ea17bc197c51b32a055ce9c
Oct 30 07:23:30.303893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.303874    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:23:30.306530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.306506    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:30.428698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.428353    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:30.441888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.441754    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:30.441888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:30.441817    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:31.066441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.066392    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" event={"ID":"10baaa70-ea0c-482a-9e67-968a8573954e","Type":"ContainerStarted","Data":"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"}
Oct 30 07:23:31.066677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.066461    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" event={"ID":"10baaa70-ea0c-482a-9e67-968a8573954e","Type":"ContainerStarted","Data":"787d00fd2a1ca82e9249a2e51091a71929db0c9721b015e5bda16b1a8ac3451b"}
Oct 30 07:23:31.066677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.066488    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:31.089721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.088723    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" event={"ID":"56b2b062-3098-48b5-b850-560c8fdaac08","Type":"ContainerStarted","Data":"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"}
Oct 30 07:23:31.089721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.088800    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" event={"ID":"56b2b062-3098-48b5-b850-560c8fdaac08","Type":"ContainerStarted","Data":"7eb5c5c93ad5ecdb37566a9b0282dc60d5ee56b58ea17bc197c51b32a055ce9c"}
Oct 30 07:23:31.089721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.089646    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:31.110464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.110411    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" podStartSLOduration=2.110395636 podStartE2EDuration="2.110395636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:31.108962259 +0000 UTC m=+1571.541726152" watchObservedRunningTime="2026-10-30 07:23:31.110395636 +0000 UTC m=+1571.543159528"
Oct 30 07:23:31.154138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.154069    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" podStartSLOduration=2.154048538 podStartE2EDuration="2.154048538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:31.150648453 +0000 UTC m=+1571.583412373" watchObservedRunningTime="2026-10-30 07:23:31.154048538 +0000 UTC m=+1571.586812432"
Oct 30 07:23:31.440004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.439960    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:31.440488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:31.440037    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:32.405045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.405008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:32.439858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.439821    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:32.440170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.440139    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:32.767043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.766958    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:23:32.808023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.807991    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:23:32.893319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.891075    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:23:32.901909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:32.901868    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:33.030382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:33.025163    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:33.441369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:33.440067    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:23:33.441369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:33.440130    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 07:23:34.444585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:34.444554    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:23:34.445725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:34.445705    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:23:34.529817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:34.529769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:36.148792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:36.148736    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:36.794287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:36.794249    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:36.937628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:36.937591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:36.968587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:36.968547    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"]
Oct 30 07:23:37.022664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.022628    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-411c9d-d7x2h"]
Oct 30 07:23:37.082362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.082249    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:37.203325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.203255    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"]
Oct 30 07:23:37.206800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.206776    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.226390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.226360    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:37.228808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.228776    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:37.231145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.231114    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"]
Oct 30 07:23:37.327191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.327157    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.327414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.327268    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.327414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.327370    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pshk\" (UniqueName: \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.429080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.429038    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.429326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.429118    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.429326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.429139    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pshk\" (UniqueName: \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.429879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.429852    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.431560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.431538    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.461609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.461574    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pshk\" (UniqueName: \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\") pod \"maas-api-revoke-sub-4640d98d-42rvv\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.487210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.487148    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:37.519088 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:37.519052    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:23:37.521412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.521390    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:37.692333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.692226    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28d5b5d-9812-419d-ae8e-000660d917ad" path="/var/lib/kubelet/pods/e28d5b5d-9812-419d-ae8e-000660d917ad/volumes"
Oct 30 07:23:37.768049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.767996    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" podUID="ce43a838-0e27-4eb0-b183-198cd490250f" containerName="payload-processing" containerID="cri-o://5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a" gracePeriod=25
Oct 30 07:23:37.808758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.808716    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" podUID="7313b357-1114-443d-a10e-de4adc88ba35" containerName="payload-pre-processing" containerID="cri-o://765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641" gracePeriod=25
Oct 30 07:23:37.833161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:37.833120    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"]
Oct 30 07:23:37.895478 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:37.895445    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fd9a3f_232e_482b_907d_6fd3ed9fceaa.slice/crio-6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b WatchSource:0}: Error finding container 6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b: Status 404 returned error can't find the container with id 6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b
Oct 30 07:23:38.099576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.099359    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:23:38.107864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.107838    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:23:38.167817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.167773    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv" event={"ID":"23fd9a3f-232e-482b-907d-6fd3ed9fceaa","Type":"ContainerStarted","Data":"6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b"}
Oct 30 07:23:38.169569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.169521    6738 generic.go:356] "Generic (PLEG): container finished" podID="7313b357-1114-443d-a10e-de4adc88ba35" containerID="765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641" exitCode=0
Oct 30 07:23:38.169745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.169570    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"
Oct 30 07:23:38.169745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.169586    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" event={"ID":"7313b357-1114-443d-a10e-de4adc88ba35","Type":"ContainerDied","Data":"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"}
Oct 30 07:23:38.169745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.169653    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls" event={"ID":"7313b357-1114-443d-a10e-de4adc88ba35","Type":"ContainerDied","Data":"c9f31b11c04e0a6dbf1a1b38fecaa318b2bd80599ee8e89b7013740b740d1610"}
Oct 30 07:23:38.169745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.169686    6738 scope.go:122] "RemoveContainer" containerID="765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"
Oct 30 07:23:38.171589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.171562    6738 generic.go:356] "Generic (PLEG): container finished" podID="ce43a838-0e27-4eb0-b183-198cd490250f" containerID="5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a" exitCode=0
Oct 30 07:23:38.171747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.171616    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"
Oct 30 07:23:38.171747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.171625    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" event={"ID":"ce43a838-0e27-4eb0-b183-198cd490250f","Type":"ContainerDied","Data":"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"}
Oct 30 07:23:38.171747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.171669    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc" event={"ID":"ce43a838-0e27-4eb0-b183-198cd490250f","Type":"ContainerDied","Data":"76eade9daa4febe177526df96a1356d79a69c262c6ccfc32ff95048ea4e448c2"}
Oct 30 07:23:38.182409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.182384    6738 scope.go:122] "RemoveContainer" containerID="765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"
Oct 30 07:23:38.182810 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:38.182772    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641\": container with ID starting with 765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641 not found: ID does not exist" containerID="765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"
Oct 30 07:23:38.182932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.182822    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641"} err="failed to get container status \"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641\": rpc error: code = NotFound desc = could not find container \"765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641\": container with ID starting with 765db7e4181052e056e21a46d92ad490bc536859dfd44b6a1cf630e5a5426641 not found: ID does not exist"
Oct 30 07:23:38.182932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.182849    6738 scope.go:122] "RemoveContainer" containerID="5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"
Oct 30 07:23:38.197096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.197070    6738 scope.go:122] "RemoveContainer" containerID="5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"
Oct 30 07:23:38.197535 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:38.197507    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a\": container with ID starting with 5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a not found: ID does not exist" containerID="5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"
Oct 30 07:23:38.197664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.197541    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a"} err="failed to get container status \"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a\": rpc error: code = NotFound desc = could not find container \"5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a\": container with ID starting with 5bd004d990592dbafca75cfbd1d4d56707cfb04fa4833e6c04d86bffc9cb847a not found: ID does not exist"
Oct 30 07:23:38.240651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.240614    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\") pod \"7313b357-1114-443d-a10e-de4adc88ba35\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") "
Oct 30 07:23:38.241067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.240720    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\" (UniqueName: \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\") pod \"ce43a838-0e27-4eb0-b183-198cd490250f\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") "
Oct 30 07:23:38.241067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.240837    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\" (UniqueName: \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\") pod \"7313b357-1114-443d-a10e-de4adc88ba35\" (UID: \"7313b357-1114-443d-a10e-de4adc88ba35\") "
Oct 30 07:23:38.241067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.240866    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\") pod \"ce43a838-0e27-4eb0-b183-198cd490250f\" (UID: \"ce43a838-0e27-4eb0-b183-198cd490250f\") "
Oct 30 07:23:38.241067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.241047    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume" pod "7313b357-1114-443d-a10e-de4adc88ba35" (UID: "7313b357-1114-443d-a10e-de4adc88ba35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:38.241329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.241269    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume" pod "ce43a838-0e27-4eb0-b183-198cd490250f" (UID: "ce43a838-0e27-4eb0-b183-198cd490250f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:38.241497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.241477    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7313b357-1114-443d-a10e-de4adc88ba35-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:38.241569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.241502    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43a838-0e27-4eb0-b183-198cd490250f-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:38.243107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.243082    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc" pod "ce43a838-0e27-4eb0-b183-198cd490250f" (UID: "ce43a838-0e27-4eb0-b183-198cd490250f"). InnerVolumeSpecName "kube-api-access-7pbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:38.243107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.243094    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg" pod "7313b357-1114-443d-a10e-de4adc88ba35" (UID: "7313b357-1114-443d-a10e-de4adc88ba35"). InnerVolumeSpecName "kube-api-access-gr9qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:38.342884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.342844    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pbjc\" (UniqueName: \"kubernetes.io/projected/ce43a838-0e27-4eb0-b183-198cd490250f-kube-api-access-7pbjc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:38.342884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.342877    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9qg\" (UniqueName: \"kubernetes.io/projected/7313b357-1114-443d-a10e-de4adc88ba35-kube-api-access-gr9qg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:38.518515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.518478    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:23:38.525536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.525500    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-45b148-bf467d78c-96pls"]
Oct 30 07:23:38.561127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.561091    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:23:38.565344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:38.565315    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-45b148-64967d8468-2gwkc"]
Oct 30 07:23:39.180718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:39.180682    6738 generic.go:356] "Generic (PLEG): container finished" podID="23fd9a3f-232e-482b-907d-6fd3ed9fceaa" containerID="3292c7370319f719a3a463519f7c2fe1495229a4e25814c8f002ca91b9be891f" exitCode=0
Oct 30 07:23:39.180943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:39.180769    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv" event={"ID":"23fd9a3f-232e-482b-907d-6fd3ed9fceaa","Type":"ContainerDied","Data":"3292c7370319f719a3a463519f7c2fe1495229a4e25814c8f002ca91b9be891f"}
Oct 30 07:23:39.691144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:39.691111    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7313b357-1114-443d-a10e-de4adc88ba35" path="/var/lib/kubelet/pods/7313b357-1114-443d-a10e-de4adc88ba35/volumes"
Oct 30 07:23:39.691612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:39.691596    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce43a838-0e27-4eb0-b183-198cd490250f" path="/var/lib/kubelet/pods/ce43a838-0e27-4eb0-b183-198cd490250f/volumes"
Oct 30 07:23:39.730080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:39.730045    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:40.335342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.335317    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:40.468798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.468710    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\") pod \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") "
Oct 30 07:23:40.468965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.468873    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\") pod \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") "
Oct 30 07:23:40.469032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.468976    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\" (UniqueName: \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\") pod \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\" (UID: \"23fd9a3f-232e-482b-907d-6fd3ed9fceaa\") "
Oct 30 07:23:40.469130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.469104    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca" pod "23fd9a3f-232e-482b-907d-6fd3ed9fceaa" (UID: "23fd9a3f-232e-482b-907d-6fd3ed9fceaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:40.469383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.469364    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:40.470955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.470930    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca" pod "23fd9a3f-232e-482b-907d-6fd3ed9fceaa" (UID: "23fd9a3f-232e-482b-907d-6fd3ed9fceaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:40.471081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.471061    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk" pod "23fd9a3f-232e-482b-907d-6fd3ed9fceaa" (UID: "23fd9a3f-232e-482b-907d-6fd3ed9fceaa"). InnerVolumeSpecName "kube-api-access-5pshk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:40.570753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.570716    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pshk\" (UniqueName: \"kubernetes.io/projected/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-kube-api-access-5pshk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:40.570753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:40.570753    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fd9a3f-232e-482b-907d-6fd3ed9fceaa-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:41.198345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:41.198311    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"
Oct 30 07:23:41.198345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:41.198330    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv" event={"ID":"23fd9a3f-232e-482b-907d-6fd3ed9fceaa","Type":"ContainerDied","Data":"6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b"}
Oct 30 07:23:41.198839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:41.198392    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ade717ce3f43cafc354ae3bcfcc3f5152828a8bede4bae2906d91a56d0a560b"
Oct 30 07:23:41.492461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:41.492361    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:41.611619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:41.611580    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:42.107637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:42.107604    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:23:43.093932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:43.093865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:43.114579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:43.114548    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:23:43.629680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:43.629647    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:43.989510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:43.989412    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:44.146944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:44.146901    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:44.341414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:44.341323    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:23:44.341635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:44.341607    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy" containerID="cri-o://6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67" gracePeriod=30
Oct 30 07:23:44.385487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:44.385432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:45.440540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:45.440497    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:46.136182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136147    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"]
Oct 30 07:23:46.136771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136756    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ce43a838-0e27-4eb0-b183-198cd490250f" containerName="payload-processing"
Oct 30 07:23:46.136852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136773    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce43a838-0e27-4eb0-b183-198cd490250f" containerName="payload-processing"
Oct 30 07:23:46.136852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136793    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7313b357-1114-443d-a10e-de4adc88ba35" containerName="payload-pre-processing"
Oct 30 07:23:46.136852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136798    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7313b357-1114-443d-a10e-de4adc88ba35" containerName="payload-pre-processing"
Oct 30 07:23:46.136852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136810    6738 cpu_manager.go:414] "containerMap: removing container" podUID="23fd9a3f-232e-482b-907d-6fd3ed9fceaa" containerName="revoke-keys"
Oct 30 07:23:46.136852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136816    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fd9a3f-232e-482b-907d-6fd3ed9fceaa" containerName="revoke-keys"
Oct 30 07:23:46.137119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136930    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce43a838-0e27-4eb0-b183-198cd490250f" containerName="payload-processing"
Oct 30 07:23:46.137119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136941    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7313b357-1114-443d-a10e-de4adc88ba35" containerName="payload-pre-processing"
Oct 30 07:23:46.137119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.136950    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fd9a3f-232e-482b-907d-6fd3ed9fceaa" containerName="revoke-keys"
Oct 30 07:23:46.139597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.139578    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.170233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.170195    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"]
Oct 30 07:23:46.177205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.177168    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:46.177429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.177232    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:46.266555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.266511    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.266766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.266602    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrwv\" (UniqueName: \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.266766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.266675    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.368012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.367974    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.368012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.368010    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrwv\" (UniqueName: \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.368254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.368029    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.368687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.368666    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.370600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.370573    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.396657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.396568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrwv\" (UniqueName: \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\") pod \"maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.457022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.456985    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:46.811307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:46.811266    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"]
Oct 30 07:23:46.812575 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:46.812545    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bd110b_4267_4b03_bec5_fc03840bac89.slice/crio-b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113 WatchSource:0}: Error finding container b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113: Status 404 returned error can't find the container with id b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113
Oct 30 07:23:47.244681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:47.244639    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6" event={"ID":"29bd110b-4267-4b03-bec5-fc03840bac89","Type":"ContainerStarted","Data":"f88003af1cc1fbf18048ebe2bb699ddb2ee59900197d4a9a2a6e7a1ee9ebf31d"}
Oct 30 07:23:47.244884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:47.244705    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6" event={"ID":"29bd110b-4267-4b03-bec5-fc03840bac89","Type":"ContainerStarted","Data":"b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113"}
Oct 30 07:23:47.285287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:47.285232    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6" podStartSLOduration=1.285218175 podStartE2EDuration="1.285218175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:47.284474674 +0000 UTC m=+1587.717238567" watchObservedRunningTime="2026-10-30 07:23:47.285218175 +0000 UTC m=+1587.717982067"
Oct 30 07:23:47.551363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:47.551326    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:48.254255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:48.254221    6738 generic.go:356] "Generic (PLEG): container finished" podID="29bd110b-4267-4b03-bec5-fc03840bac89" containerID="f88003af1cc1fbf18048ebe2bb699ddb2ee59900197d4a9a2a6e7a1ee9ebf31d" exitCode=0
Oct 30 07:23:48.254468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:48.254328    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6" event={"ID":"29bd110b-4267-4b03-bec5-fc03840bac89","Type":"ContainerDied","Data":"f88003af1cc1fbf18048ebe2bb699ddb2ee59900197d4a9a2a6e7a1ee9ebf31d"}
Oct 30 07:23:49.236972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.236931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:49.351222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.351190    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:49.476642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.476614    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:49.630410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.629922    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\") pod \"29bd110b-4267-4b03-bec5-fc03840bac89\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") "
Oct 30 07:23:49.632013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.631987    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\") pod \"29bd110b-4267-4b03-bec5-fc03840bac89\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") "
Oct 30 07:23:49.635844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.632793    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca" pod "29bd110b-4267-4b03-bec5-fc03840bac89" (UID: "29bd110b-4267-4b03-bec5-fc03840bac89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:49.635844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.635663    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca" pod "29bd110b-4267-4b03-bec5-fc03840bac89" (UID: "29bd110b-4267-4b03-bec5-fc03840bac89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:49.635844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.635732    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:49.636509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.636481    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\" (UniqueName: \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\") pod \"29bd110b-4267-4b03-bec5-fc03840bac89\" (UID: \"29bd110b-4267-4b03-bec5-fc03840bac89\") "
Oct 30 07:23:49.637900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.637145    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.637900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.637166    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29bd110b-4267-4b03-bec5-fc03840bac89-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.651570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.649561    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv" pod "29bd110b-4267-4b03-bec5-fc03840bac89" (UID: "29bd110b-4267-4b03-bec5-fc03840bac89"). InnerVolumeSpecName "kube-api-access-qcrwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:49.659362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.659322    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:23:49.659582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.659551    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy" containerID="cri-o://d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821" gracePeriod=30
Oct 30 07:23:49.678743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.678717    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:23:49.711750 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:49.711580    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bd110b_4267_4b03_bec5_fc03840bac89.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 07:23:49.711750 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:49.711651    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bd110b_4267_4b03_bec5_fc03840bac89.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 07:23:49.739028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.738986    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrwv\" (UniqueName: \"kubernetes.io/projected/29bd110b-4267-4b03-bec5-fc03840bac89-kube-api-access-qcrwv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.786926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.786836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:49.845701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.843531    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846054    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846157    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846222    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846285    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846398    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846452    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846483    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.846654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.846556    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\") pod \"63a45d26-7fac-475b-91a4-c305f67aa821\" (UID: \"63a45d26-7fac-475b-91a4-c305f67aa821\") "
Oct 30 07:23:49.848181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.847949    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:49.878865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.878816    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:49.880513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.880480    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:49.880956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.880932    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:49.883707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.881441    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:49.884084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.882782    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:49.884401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.884380    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "kube-api-access-9mwjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:49.889478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.889441    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:49.893479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.893445    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token" pod "63a45d26-7fac-475b-91a4-c305f67aa821" (UID: "63a45d26-7fac-475b-91a4-c305f67aa821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:49.948671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948633    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a45d26-7fac-475b-91a4-c305f67aa821-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948664    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948675    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948684    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948693    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a45d26-7fac-475b-91a4-c305f67aa821-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948706    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948717    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948724    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a45d26-7fac-475b-91a4-c305f67aa821-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:49.948947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:49.948732    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwjw\" (UniqueName: \"kubernetes.io/projected/63a45d26-7fac-475b-91a4-c305f67aa821-kube-api-access-9mwjw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:50.223223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223190    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"]
Oct 30 07:23:50.223870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223849    6738 cpu_manager.go:414] "containerMap: removing container" podUID="29bd110b-4267-4b03-bec5-fc03840bac89" containerName="revoke-keys"
Oct 30 07:23:50.223870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223869    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bd110b-4267-4b03-bec5-fc03840bac89" containerName="revoke-keys"
Oct 30 07:23:50.224013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223895    6738 cpu_manager.go:414] "containerMap: removing container" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:50.224013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223901    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:50.224013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223919    6738 cpu_manager.go:414] "containerMap: removing container" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:50.224013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.223924    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:50.224013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.224004    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:50.224177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.224018    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bd110b-4267-4b03-bec5-fc03840bac89" containerName="revoke-keys"
Oct 30 07:23:50.227049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.227033    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.245605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.245577    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"]
Oct 30 07:23:50.274153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.274121    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"
Oct 30 07:23:50.274153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.274135    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6" event={"ID":"29bd110b-4267-4b03-bec5-fc03840bac89","Type":"ContainerDied","Data":"b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113"}
Oct 30 07:23:50.274458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.274194    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e53e502a94f783d3fcdb33b469a0d834673d369aa03cc0f322f11e596df113"
Oct 30 07:23:50.276516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.276484    6738 generic.go:356] "Generic (PLEG): container finished" podID="63a45d26-7fac-475b-91a4-c305f67aa821" containerID="6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67" exitCode=0
Oct 30 07:23:50.276756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.276667    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerDied","Data":"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"}
Oct 30 07:23:50.276756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.276712    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl" event={"ID":"63a45d26-7fac-475b-91a4-c305f67aa821","Type":"ContainerDied","Data":"dc0b204ee2b841dd90beda1f19ef2bcff3b550d30831221b926926ce3432a036"}
Oct 30 07:23:50.276756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.276739    6738 scope.go:122] "RemoveContainer" containerID="6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"
Oct 30 07:23:50.277054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.276922    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"
Oct 30 07:23:50.294560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.294537    6738 scope.go:122] "RemoveContainer" containerID="f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"
Oct 30 07:23:50.305055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.305033    6738 scope.go:122] "RemoveContainer" containerID="6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"
Oct 30 07:23:50.305397 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:50.305371    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67\": container with ID starting with 6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67 not found: ID does not exist" containerID="6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"
Oct 30 07:23:50.305462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.305409    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67"} err="failed to get container status \"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67\": rpc error: code = NotFound desc = could not find container \"6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67\": container with ID starting with 6d5813462a6a51f0ec2e22bd98dd9344303de9ccae1f81b6e59681fb61770e67 not found: ID does not exist"
Oct 30 07:23:50.305462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.305430    6738 scope.go:122] "RemoveContainer" containerID="f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"
Oct 30 07:23:50.305715 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:50.305689    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f\": container with ID starting with f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f not found: ID does not exist" containerID="f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"
Oct 30 07:23:50.305825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.305720    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f"} err="failed to get container status \"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f\": rpc error: code = NotFound desc = could not find container \"f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f\": container with ID starting with f9151c0b54fccaddeb3b526b12cd2914f1c1827d7f0a2261432a2cf2ad58928f not found: ID does not exist"
Oct 30 07:23:50.333130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.333098    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:23:50.342363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.342331    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-411c9d-openshift-default-6f7c85d8b5-sp7wl"]
Oct 30 07:23:50.354633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.354602    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.354826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.354728    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgtw\" (UniqueName: \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.354902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.354832    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.456612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.456574    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.456802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.456659    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgtw\" (UniqueName: \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.456802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.456742    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.457440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.457413    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.459149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.459127    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.494334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.494235    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgtw\" (UniqueName: \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\") pod \"maas-api-revoke-sub-4a9f8fc8-4xb5c\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.540352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.540278    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:50.846941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:50.846911    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"]
Oct 30 07:23:50.848209 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:50.848178    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24131829_0c28_4fdc_bd4e_e12db24c6e85.slice/crio-6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39 WatchSource:0}: Error finding container 6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39: Status 404 returned error can't find the container with id 6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39
Oct 30 07:23:51.286241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:51.286202    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c" event={"ID":"24131829-0c28-4fdc-bd4e-e12db24c6e85","Type":"ContainerStarted","Data":"bcd255555ae94d0c2272bff352a441ae9b983382a2c9e7197157ee3f822754d7"}
Oct 30 07:23:51.286682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:51.286268    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c" event={"ID":"24131829-0c28-4fdc-bd4e-e12db24c6e85","Type":"ContainerStarted","Data":"6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39"}
Oct 30 07:23:51.335439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:51.335384    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c" podStartSLOduration=1.335368527 podStartE2EDuration="1.335368527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:51.332498086 +0000 UTC m=+1591.765261978" watchObservedRunningTime="2026-10-30 07:23:51.335368527 +0000 UTC m=+1591.768132419"
Oct 30 07:23:51.501072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:51.500968    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:51.690789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:51.690752    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" path="/var/lib/kubelet/pods/63a45d26-7fac-475b-91a4-c305f67aa821/volumes"
Oct 30 07:23:52.296767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.296728    6738 generic.go:356] "Generic (PLEG): container finished" podID="24131829-0c28-4fdc-bd4e-e12db24c6e85" containerID="bcd255555ae94d0c2272bff352a441ae9b983382a2c9e7197157ee3f822754d7" exitCode=0
Oct 30 07:23:52.297191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.296816    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c" event={"ID":"24131829-0c28-4fdc-bd4e-e12db24c6e85","Type":"ContainerDied","Data":"bcd255555ae94d0c2272bff352a441ae9b983382a2c9e7197157ee3f822754d7"}
Oct 30 07:23:52.420360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.420316    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:23:52.421058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.421039    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a45d26-7fac-475b-91a4-c305f67aa821" containerName="istio-proxy"
Oct 30 07:23:52.423674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.423653    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.442183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.441853    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb6be4-openshift-default-dockercfg-2kstl\""
Oct 30 07:23:52.461739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.461697    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:23:52.585324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585202    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585422    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585463    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585487    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585549    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585610    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585718    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9l6\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585772    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.585833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.585798    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.686966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.686916    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.686979    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687007    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687043    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687120    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687146    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9l6\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687176    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687217    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687322    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687423    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687506    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687556    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.687906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.687752    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.688103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.688081    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.689514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.689490    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.689719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.689704    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.711024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.710990    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.711252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.711225    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9l6\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:52.737556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:52.737517    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:53.057499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.049518    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"]
Oct 30 07:23:53.057499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.054123    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cfdfb8-cgmhm"]
Oct 30 07:23:53.094348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.094314    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:23:53.102388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.100816    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:53.102388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.100925    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:53.102388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.100970    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:23:53.307817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.307713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" event={"ID":"3ea59724-25db-44f8-b816-a6e85bf97b7c","Type":"ContainerStarted","Data":"26911ed328b001486e80932fd8b79e9f445f6b05efcbf638c3fc76b47630e362"}
Oct 30 07:23:53.307817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.307784    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" event={"ID":"3ea59724-25db-44f8-b816-a6e85bf97b7c","Type":"ContainerStarted","Data":"ca8f4544488871daa170ef8f78f070ae3c38f4354918eb123365aa0c48a71162"}
Oct 30 07:23:53.361324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.361242    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podStartSLOduration=1.361218665 podStartE2EDuration="1.361218665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:53.357543138 +0000 UTC m=+1593.790307030" watchObservedRunningTime="2026-10-30 07:23:53.361218665 +0000 UTC m=+1593.793982562"
Oct 30 07:23:53.452599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.452569    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:53.599675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.599576    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\") pod \"24131829-0c28-4fdc-bd4e-e12db24c6e85\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") "
Oct 30 07:23:53.599868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.599847    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\" (UniqueName: \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\") pod \"24131829-0c28-4fdc-bd4e-e12db24c6e85\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") "
Oct 30 07:23:53.599918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.599903    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\") pod \"24131829-0c28-4fdc-bd4e-e12db24c6e85\" (UID: \"24131829-0c28-4fdc-bd4e-e12db24c6e85\") "
Oct 30 07:23:53.599961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.599926    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca" pod "24131829-0c28-4fdc-bd4e-e12db24c6e85" (UID: "24131829-0c28-4fdc-bd4e-e12db24c6e85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:53.600401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.600378    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:53.601982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.601959    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca" pod "24131829-0c28-4fdc-bd4e-e12db24c6e85" (UID: "24131829-0c28-4fdc-bd4e-e12db24c6e85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:53.602103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.602075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw" pod "24131829-0c28-4fdc-bd4e-e12db24c6e85" (UID: "24131829-0c28-4fdc-bd4e-e12db24c6e85"). InnerVolumeSpecName "kube-api-access-tvgtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:53.701574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.701535    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgtw\" (UniqueName: \"kubernetes.io/projected/24131829-0c28-4fdc-bd4e-e12db24c6e85-kube-api-access-tvgtw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:53.701574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.701567    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24131829-0c28-4fdc-bd4e-e12db24c6e85-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:53.704490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.704454    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e02cca8-5dda-4d57-8a75-c95badd2dab9" path="/var/lib/kubelet/pods/0e02cca8-5dda-4d57-8a75-c95badd2dab9/volumes"
Oct 30 07:23:53.738608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:53.738573    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:23:54.096319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.091459    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:54.316284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.316255    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"
Oct 30 07:23:54.316284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.316289    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c" event={"ID":"24131829-0c28-4fdc-bd4e-e12db24c6e85","Type":"ContainerDied","Data":"6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39"}
Oct 30 07:23:54.316908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.316351    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d90088d1b20c9827e0fdbf81e65c539105d45e3dd9db6f4dbe61b6351b11f39"
Oct 30 07:23:54.739646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.739607    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:54.739821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:54.739673    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:55.004564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.004538    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:23:55.118020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.117982    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118038    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118159    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118188    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118210    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118240    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118320    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118328    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:55.118546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118361    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118505    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\") pod \"1871aeae-1fad-4a35-ad48-4f4c82111d52\" (UID: \"1871aeae-1fad-4a35-ad48-4f4c82111d52\") "
Oct 30 07:23:55.118546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118517    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:55.118719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118618    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:55.118719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118689    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:55.118924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.118901    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:55.119138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.119119    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1871aeae-1fad-4a35-ad48-4f4c82111d52-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.119215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.119143    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.119215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.119160    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.119215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.119175    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.119215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.119191    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.120605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.120582    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:55.120728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.120665    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "kube-api-access-vxvnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:55.120885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.120869    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:55.120934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.120886    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy" pod "1871aeae-1fad-4a35-ad48-4f4c82111d52" (UID: "1871aeae-1fad-4a35-ad48-4f4c82111d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:55.220902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.220801    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvnj\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-kube-api-access-vxvnj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.220902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.220843    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.220902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.220856    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.220902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.220866    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1871aeae-1fad-4a35-ad48-4f4c82111d52-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:55.335422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.335383    6738 generic.go:356] "Generic (PLEG): container finished" podID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerID="d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821" exitCode=0
Oct 30 07:23:55.335833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.335461    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"
Oct 30 07:23:55.335833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.335461    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerDied","Data":"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"}
Oct 30 07:23:55.335833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.335622    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt" event={"ID":"1871aeae-1fad-4a35-ad48-4f4c82111d52","Type":"ContainerDied","Data":"4f89c82200f211552da92279e76453ac8fe8f412ac456baf0eacd4aceb05fe57"}
Oct 30 07:23:55.335833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.335651    6738 scope.go:122] "RemoveContainer" containerID="d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"
Oct 30 07:23:55.346998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.346979    6738 scope.go:122] "RemoveContainer" containerID="ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"
Oct 30 07:23:55.360315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.360278    6738 scope.go:122] "RemoveContainer" containerID="d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"
Oct 30 07:23:55.360670 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:55.360647    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821\": container with ID starting with d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821 not found: ID does not exist" containerID="d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"
Oct 30 07:23:55.360736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.360681    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821"} err="failed to get container status \"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821\": rpc error: code = NotFound desc = could not find container \"d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821\": container with ID starting with d882ae7e1e42ddccac33f12604ea3df1e9f5a009b8fced76c3678b8387c09821 not found: ID does not exist"
Oct 30 07:23:55.360736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.360701    6738 scope.go:122] "RemoveContainer" containerID="ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"
Oct 30 07:23:55.360978 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:55.360958    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1\": container with ID starting with ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1 not found: ID does not exist" containerID="ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"
Oct 30 07:23:55.361031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.360986    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1"} err="failed to get container status \"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1\": rpc error: code = NotFound desc = could not find container \"ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1\": container with ID starting with ea9ee70b58b56fa62ffc2bc3d91561d25eb84cafcc42188f3c00c4f41d229bb1 not found: ID does not exist"
Oct 30 07:23:55.399905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.399868    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:23:55.412581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.412542    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a51baa-openshift-default-65b9d7cc89-b5nzt"]
Oct 30 07:23:55.444146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444112    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"]
Oct 30 07:23:55.444782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444761    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.444895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444783    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.444895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444809    6738 cpu_manager.go:414] "containerMap: removing container" podUID="24131829-0c28-4fdc-bd4e-e12db24c6e85" containerName="revoke-keys"
Oct 30 07:23:55.444895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444816    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24131829-0c28-4fdc-bd4e-e12db24c6e85" containerName="revoke-keys"
Oct 30 07:23:55.444895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444838    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.444895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.444846    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.445225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.445011    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.445225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.445027    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="24131829-0c28-4fdc-bd4e-e12db24c6e85" containerName="revoke-keys"
Oct 30 07:23:55.445225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.445038    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" containerName="istio-proxy"
Oct 30 07:23:55.448364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.448342    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.463609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.463584    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:55.463861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.463837    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:55.466014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.465989    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"]
Oct 30 07:23:55.625491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.625461    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.625673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.625515    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzr4h\" (UniqueName: \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.625673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.625651    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.700283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.700251    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1871aeae-1fad-4a35-ad48-4f4c82111d52" path="/var/lib/kubelet/pods/1871aeae-1fad-4a35-ad48-4f4c82111d52/volumes"
Oct 30 07:23:55.726947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.726913    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.727109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.726970    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzr4h\" (UniqueName: \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.727109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.727045    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.727640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.727601    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.730123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.730093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.739006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.737877    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:55.739006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.737925    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:55.751765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.751727    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzr4h\" (UniqueName: \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.766443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.766058    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:55.801327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:55.801237    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:56.146913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.146831    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"]
Oct 30 07:23:56.155579 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:23:56.155535    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c241b8_53f4_46cd_a9d7_47dd0729da47.slice/crio-a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae WatchSource:0}: Error finding container a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae: Status 404 returned error can't find the container with id a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae
Oct 30 07:23:56.363315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.362491    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8" event={"ID":"64c241b8-53f4-46cd-a9d7-47dd0729da47","Type":"ContainerStarted","Data":"a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae"}
Oct 30 07:23:56.738323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.738252    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:56.738530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.738346    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:56.738678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.738606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:56.889353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.888765    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:56.955907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.955865    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"]
Oct 30 07:23:56.988453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:56.985549    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-45b148-plc2c"]
Oct 30 07:23:57.004049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.004002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:57.399902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.399852    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8" event={"ID":"64c241b8-53f4-46cd-a9d7-47dd0729da47","Type":"ContainerStarted","Data":"6cce4611c0aae78ba5a2c93763030d6edd881c66d062a1a4b5487e5704c1186a"}
Oct 30 07:23:57.442937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.442879    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8" podStartSLOduration=2.442864889 podStartE2EDuration="2.442864889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:57.439978043 +0000 UTC m=+1597.872741936" watchObservedRunningTime="2026-10-30 07:23:57.442864889 +0000 UTC m=+1597.875628816"
Oct 30 07:23:57.466763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.466731    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:57.699865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.699778    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aff3db-424d-4e18-bcef-a27244fb3f6c" path="/var/lib/kubelet/pods/e5aff3db-424d-4e18-bcef-a27244fb3f6c/volumes"
Oct 30 07:23:57.738403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.738359    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:57.738601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:57.738420    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:58.408699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.408667    6738 generic.go:356] "Generic (PLEG): container finished" podID="64c241b8-53f4-46cd-a9d7-47dd0729da47" containerID="6cce4611c0aae78ba5a2c93763030d6edd881c66d062a1a4b5487e5704c1186a" exitCode=0
Oct 30 07:23:58.409128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.408754    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8" event={"ID":"64c241b8-53f4-46cd-a9d7-47dd0729da47","Type":"ContainerDied","Data":"6cce4611c0aae78ba5a2c93763030d6edd881c66d062a1a4b5487e5704c1186a"}
Oct 30 07:23:58.641624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.641570    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:58.738890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.738805    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:58.738890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.738868    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:58.766188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.766155    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:23:58.766380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.766355    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" podUID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" containerName="payload-processing" containerID="cri-o://c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f" gracePeriod=30
Oct 30 07:23:58.829330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.829275    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:23:58.830095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.829974    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" podUID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" containerName="payload-pre-processing" containerID="cri-o://2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750" gracePeriod=30
Oct 30 07:23:58.863957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:58.863904    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:59.200415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.200370    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:59.292112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.291782    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:23:59.313915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.313889    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:23:59.333363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.333334    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\" (UniqueName: \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\") pod \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") "
Oct 30 07:23:59.333551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.333427    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\") pod \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\" (UID: \"7bf2d105-82e6-4464-a75f-9d484e89b3b6\") "
Oct 30 07:23:59.333774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.333752    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume" pod "7bf2d105-82e6-4464-a75f-9d484e89b3b6" (UID: "7bf2d105-82e6-4464-a75f-9d484e89b3b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:59.335701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.335672    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7" pod "7bf2d105-82e6-4464-a75f-9d484e89b3b6" (UID: "7bf2d105-82e6-4464-a75f-9d484e89b3b6"). InnerVolumeSpecName "kube-api-access-wjbc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:59.413274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.413237    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:59.419620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.419586    6738 generic.go:356] "Generic (PLEG): container finished" podID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" containerID="2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750" exitCode=0
Oct 30 07:23:59.419763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.419643    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"
Oct 30 07:23:59.419763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.419666    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" event={"ID":"b71bc68b-2bcd-4118-a61f-25516b5e31cc","Type":"ContainerDied","Data":"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"}
Oct 30 07:23:59.419763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.419733    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm" event={"ID":"b71bc68b-2bcd-4118-a61f-25516b5e31cc","Type":"ContainerDied","Data":"32c351baa3159dd9cd0132a0aa78be8d55f376b919884244bb506938dcdf5771"}
Oct 30 07:23:59.419923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.419762    6738 scope.go:122] "RemoveContainer" containerID="2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"
Oct 30 07:23:59.421587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.421564    6738 generic.go:356] "Generic (PLEG): container finished" podID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" containerID="c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f" exitCode=0
Oct 30 07:23:59.421682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.421601    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" event={"ID":"7bf2d105-82e6-4464-a75f-9d484e89b3b6","Type":"ContainerDied","Data":"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"}
Oct 30 07:23:59.421682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.421629    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"
Oct 30 07:23:59.421682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.421655    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj" event={"ID":"7bf2d105-82e6-4464-a75f-9d484e89b3b6","Type":"ContainerDied","Data":"ae632622cd2e5dcd6335558a7afda724d9c5b367de17eeac3c388cea3484e5ae"}
Oct 30 07:23:59.434284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.434258    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\" (UniqueName: \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\") pod \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") "
Oct 30 07:23:59.434284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.434354    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\") pod \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\" (UID: \"b71bc68b-2bcd-4118-a61f-25516b5e31cc\") "
Oct 30 07:23:59.434634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.434575    6738 scope.go:122] "RemoveContainer" containerID="2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.434781    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume" pod "b71bc68b-2bcd-4118-a61f-25516b5e31cc" (UID: "b71bc68b-2bcd-4118-a61f-25516b5e31cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.435017    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71bc68b-2bcd-4118-a61f-25516b5e31cc-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.435035    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbc7\" (UniqueName: \"kubernetes.io/projected/7bf2d105-82e6-4464-a75f-9d484e89b3b6-kube-api-access-wjbc7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.435044    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf2d105-82e6-4464-a75f-9d484e89b3b6-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:59.435112    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750\": container with ID starting with 2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750 not found: ID does not exist" containerID="2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.435156    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750"} err="failed to get container status \"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750\": rpc error: code = NotFound desc = could not find container \"2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750\": container with ID starting with 2f0a611f8e4977d2a1084dfc769b3df9135ea75c49b3b858b4a34f69db442750 not found: ID does not exist"
Oct 30 07:23:59.435373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.435182    6738 scope.go:122] "RemoveContainer" containerID="c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"
Oct 30 07:23:59.437040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.437015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7" pod "b71bc68b-2bcd-4118-a61f-25516b5e31cc" (UID: "b71bc68b-2bcd-4118-a61f-25516b5e31cc"). InnerVolumeSpecName "kube-api-access-s8mn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:59.447620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.447595    6738 scope.go:122] "RemoveContainer" containerID="c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"
Oct 30 07:23:59.448017 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:23:59.447991    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f\": container with ID starting with c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f not found: ID does not exist" containerID="c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"
Oct 30 07:23:59.448069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.448027    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f"} err="failed to get container status \"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f\": rpc error: code = NotFound desc = could not find container \"c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f\": container with ID starting with c75e15b3a2e751029cd2207b92e069bab4e9fbf8e3480a003a3fdd93be9a9d6f not found: ID does not exist"
Oct 30 07:23:59.474227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.474197    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:23:59.480810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.480466    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8d44bf-f8fdd9874-j48qj"]
Oct 30 07:23:59.536618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.536582    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mn7\" (UniqueName: \"kubernetes.io/projected/b71bc68b-2bcd-4118-a61f-25516b5e31cc-kube-api-access-s8mn7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.559838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.559809    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"]
Oct 30 07:23:59.561659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.561623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:59.563701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.563672    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8d44bf-jf8f6"]
Oct 30 07:23:59.566669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.566644    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:23:59.644321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.639440    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\") pod \"64c241b8-53f4-46cd-a9d7-47dd0729da47\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") "
Oct 30 07:23:59.644321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.639552    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\" (UniqueName: \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\") pod \"64c241b8-53f4-46cd-a9d7-47dd0729da47\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") "
Oct 30 07:23:59.644321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.639630    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\") pod \"64c241b8-53f4-46cd-a9d7-47dd0729da47\" (UID: \"64c241b8-53f4-46cd-a9d7-47dd0729da47\") "
Oct 30 07:23:59.667194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.667113    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca" pod "64c241b8-53f4-46cd-a9d7-47dd0729da47" (UID: "64c241b8-53f4-46cd-a9d7-47dd0729da47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:59.668199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.668166    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca" pod "64c241b8-53f4-46cd-a9d7-47dd0729da47" (UID: "64c241b8-53f4-46cd-a9d7-47dd0729da47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:59.670461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.670435    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h" pod "64c241b8-53f4-46cd-a9d7-47dd0729da47" (UID: "64c241b8-53f4-46cd-a9d7-47dd0729da47"). InnerVolumeSpecName "kube-api-access-hzr4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:59.679008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.678966    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:23:59.694231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.694196    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bd110b-4267-4b03-bec5-fc03840bac89" path="/var/lib/kubelet/pods/29bd110b-4267-4b03-bec5-fc03840bac89/volumes"
Oct 30 07:23:59.694867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.694847    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" path="/var/lib/kubelet/pods/7bf2d105-82e6-4464-a75f-9d484e89b3b6/volumes"
Oct 30 07:23:59.738820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.738781    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 07:23:59.739015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.738861    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 07:23:59.742010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.741979    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.742010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.742012    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c241b8-53f4-46cd-a9d7-47dd0729da47-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.742179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.742025    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzr4h\" (UniqueName: \"kubernetes.io/projected/64c241b8-53f4-46cd-a9d7-47dd0729da47-kube-api-access-hzr4h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:23:59.801955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.801921    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:23:59.832874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:23:59.832787    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8d44bf-7fd86f86bc-mlhpm"]
Oct 30 07:24:00.133534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:00.133484    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:00.433320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:00.433194    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"
Oct 30 07:24:00.433320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:00.433229    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8" event={"ID":"64c241b8-53f4-46cd-a9d7-47dd0729da47","Type":"ContainerDied","Data":"a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae"}
Oct 30 07:24:00.433919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:00.433289    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69af424912434dca72cd6a182f7004d2d4eeefedeeaa644017b23d492cd3bae"
Oct 30 07:24:01.075455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:01.075417    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:24:01.076175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:01.076141    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:24:01.077279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:01.077255    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:24:01.690445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:01.690382    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" path="/var/lib/kubelet/pods/b71bc68b-2bcd-4118-a61f-25516b5e31cc/volumes"
Oct 30 07:24:01.760726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:01.760689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:03.126991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.126951    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"]
Oct 30 07:24:03.127770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127747    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" containerName="payload-processing"
Oct 30 07:24:03.127879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127772    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" containerName="payload-processing"
Oct 30 07:24:03.127879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127795    6738 cpu_manager.go:414] "containerMap: removing container" podUID="64c241b8-53f4-46cd-a9d7-47dd0729da47" containerName="revoke-keys"
Oct 30 07:24:03.127879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127803    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c241b8-53f4-46cd-a9d7-47dd0729da47" containerName="revoke-keys"
Oct 30 07:24:03.127879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127816    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" containerName="payload-pre-processing"
Oct 30 07:24:03.127879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.127825    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" containerName="payload-pre-processing"
Oct 30 07:24:03.128145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.128003    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf2d105-82e6-4464-a75f-9d484e89b3b6" containerName="payload-processing"
Oct 30 07:24:03.128145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.128019    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c241b8-53f4-46cd-a9d7-47dd0729da47" containerName="revoke-keys"
Oct 30 07:24:03.128145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.128032    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71bc68b-2bcd-4118-a61f-25516b5e31cc" containerName="payload-pre-processing"
Oct 30 07:24:03.131584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.131561    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.150048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.150021    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"]
Oct 30 07:24:03.157035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.156995    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:03.157243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.157019    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:03.188438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.188403    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.188438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.188439    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95fw\" (UniqueName: \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.188709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.188651    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.290467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.290430    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.290677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.290473    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95fw\" (UniqueName: \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.290677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.290606    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.291247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.291225    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.292846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.292817    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.317564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.317533    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95fw\" (UniqueName: \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\") pod \"maas-api-revoke-sub-a4327f18-7xd55\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.449763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.449675    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:03.682206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.682165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:03.755533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.755502    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"]
Oct 30 07:24:03.756364 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:03.756327    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366184e_cb74_4f47_9d7a_afc8102053ea.slice/crio-f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc WatchSource:0}: Error finding container f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc: Status 404 returned error can't find the container with id f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc
Oct 30 07:24:03.834399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:03.831725    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:04.097088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.097057    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:24:04.097460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.097258    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy" containerID="cri-o://f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b" gracePeriod=30
Oct 30 07:24:04.133337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.133261    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:04.465851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.465816    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55" event={"ID":"7366184e-cb74-4f47-9d7a-afc8102053ea","Type":"ContainerStarted","Data":"3c9753a433264b3468898acbaa257f10b540a48652e9438419920452ca59d704"}
Oct 30 07:24:04.466048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.465873    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55" event={"ID":"7366184e-cb74-4f47-9d7a-afc8102053ea","Type":"ContainerStarted","Data":"f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc"}
Oct 30 07:24:04.509534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.509479    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55" podStartSLOduration=1.509462522 podStartE2EDuration="1.509462522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:04.505924713 +0000 UTC m=+1604.938688607" watchObservedRunningTime="2026-10-30 07:24:04.509462522 +0000 UTC m=+1604.942226692"
Oct 30 07:24:04.718823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.718745    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"]
Oct 30 07:24:04.721992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.721971    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.741330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.740185    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"]
Oct 30 07:24:04.825387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.825353    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.825740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.825521    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.825740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.825694    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmfp\" (UniqueName: \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.927152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.927112    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmfp\" (UniqueName: \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.927376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.927170    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.927376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.927268    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.927962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.927932    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.929762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.929740    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:04.950811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:04.950780    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmfp\" (UniqueName: \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\") pod \"maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:05.066886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.066799    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:05.375775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.375731    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"]
Oct 30 07:24:05.376491 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:05.376464    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31ce3c0_821e_470f_9e38_b6c510825b81.slice/crio-b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a WatchSource:0}: Error finding container b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a: Status 404 returned error can't find the container with id b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a
Oct 30 07:24:05.475060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.475025    6738 generic.go:356] "Generic (PLEG): container finished" podID="7366184e-cb74-4f47-9d7a-afc8102053ea" containerID="3c9753a433264b3468898acbaa257f10b540a48652e9438419920452ca59d704" exitCode=0
Oct 30 07:24:05.475257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.475064    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55" event={"ID":"7366184e-cb74-4f47-9d7a-afc8102053ea","Type":"ContainerDied","Data":"3c9753a433264b3468898acbaa257f10b540a48652e9438419920452ca59d704"}
Oct 30 07:24:05.476607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.476574    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f" event={"ID":"f31ce3c0-821e-470f-9e38-b6c510825b81","Type":"ContainerStarted","Data":"b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a"}
Oct 30 07:24:05.631161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.631066    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:05.972721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:05.972642    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:24:06.007220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.007175    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:24:06.132879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.132841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:06.247369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.247025    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:06.484987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.484953    6738 generic.go:356] "Generic (PLEG): container finished" podID="f31ce3c0-821e-470f-9e38-b6c510825b81" containerID="79a998056639e24432acee15dbb9a4fb3f7946c361794b1a51b75295620b0804" exitCode=0
Oct 30 07:24:06.485404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.485042    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f" event={"ID":"f31ce3c0-821e-470f-9e38-b6c510825b81","Type":"ContainerDied","Data":"79a998056639e24432acee15dbb9a4fb3f7946c361794b1a51b75295620b0804"}
Oct 30 07:24:06.755339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.755311    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:06.872344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.872281    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\") pod \"7366184e-cb74-4f47-9d7a-afc8102053ea\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") "
Oct 30 07:24:06.872553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.872486    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\") pod \"7366184e-cb74-4f47-9d7a-afc8102053ea\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") "
Oct 30 07:24:06.872635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.872600    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\" (UniqueName: \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\") pod \"7366184e-cb74-4f47-9d7a-afc8102053ea\" (UID: \"7366184e-cb74-4f47-9d7a-afc8102053ea\") "
Oct 30 07:24:06.872971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.872940    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca" pod "7366184e-cb74-4f47-9d7a-afc8102053ea" (UID: "7366184e-cb74-4f47-9d7a-afc8102053ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:06.873615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.873588    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:06.874782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.874755    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw" pod "7366184e-cb74-4f47-9d7a-afc8102053ea" (UID: "7366184e-cb74-4f47-9d7a-afc8102053ea"). InnerVolumeSpecName "kube-api-access-k95fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:06.875021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.874999    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca" pod "7366184e-cb74-4f47-9d7a-afc8102053ea" (UID: "7366184e-cb74-4f47-9d7a-afc8102053ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:06.975136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.975103    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95fw\" (UniqueName: \"kubernetes.io/projected/7366184e-cb74-4f47-9d7a-afc8102053ea-kube-api-access-k95fw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:06.975136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:06.975133    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366184e-cb74-4f47-9d7a-afc8102053ea-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:07.147529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.147439    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:07.264143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.264107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:07.494686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.494609    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"
Oct 30 07:24:07.494686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.494623    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55" event={"ID":"7366184e-cb74-4f47-9d7a-afc8102053ea","Type":"ContainerDied","Data":"f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc"}
Oct 30 07:24:07.494686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.494680    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cdf261232a98409067c372bb6806b94e9a72e8b068fafb83fa0cc595c6e4bc"
Oct 30 07:24:07.528617 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:07.528575    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:24:07.587340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.583711    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:24:07.587340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.584849    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7366184e-cb74-4f47-9d7a-afc8102053ea" containerName="revoke-keys"
Oct 30 07:24:07.587340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.584870    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366184e-cb74-4f47-9d7a-afc8102053ea" containerName="revoke-keys"
Oct 30 07:24:07.587340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.585077    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366184e-cb74-4f47-9d7a-afc8102053ea" containerName="revoke-keys"
Oct 30 07:24:07.591345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.590224    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.620847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.620809    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:24:07.630572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.630539    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:24:07.630867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.630841    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" containerID="cri-o://7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4" gracePeriod=30
Oct 30 07:24:07.661281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.661250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:07.685053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685021    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685074    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685186    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685331    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685446    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685515    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685548    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685573    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d7g\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.685709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.685680    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.696983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.696955    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:07.786852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.786748    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\") pod \"f31ce3c0-821e-470f-9e38-b6c510825b81\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") "
Oct 30 07:24:07.787048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.786873    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\") pod \"f31ce3c0-821e-470f-9e38-b6c510825b81\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") "
Oct 30 07:24:07.787048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.786911    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\" (UniqueName: \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\") pod \"f31ce3c0-821e-470f-9e38-b6c510825b81\" (UID: \"f31ce3c0-821e-470f-9e38-b6c510825b81\") "
Oct 30 07:24:07.787237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.787340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787285    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca" pod "f31ce3c0-821e-470f-9e38-b6c510825b81" (UID: "f31ce3c0-821e-470f-9e38-b6c510825b81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:07.787418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787344    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.787418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787380    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787412    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787457    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d7g\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787486    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787624    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787671    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787707    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787818    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.787813    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.788074    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.788201    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.788418    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.788807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.788502    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.789692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.789665    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp" pod "f31ce3c0-821e-470f-9e38-b6c510825b81" (UID: "f31ce3c0-821e-470f-9e38-b6c510825b81"). InnerVolumeSpecName "kube-api-access-jnmfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:07.789942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.789921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca" pod "f31ce3c0-821e-470f-9e38-b6c510825b81" (UID: "f31ce3c0-821e-470f-9e38-b6c510825b81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:07.790630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.790606    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.790906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.790887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.795234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.795205    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:07.811554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.811522    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.811694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.811657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d7g\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:07.893315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.890503    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmfp\" (UniqueName: \"kubernetes.io/projected/f31ce3c0-821e-470f-9e38-b6c510825b81-kube-api-access-jnmfp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:07.893315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.890541    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ce3c0-821e-470f-9e38-b6c510825b81-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:07.926867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:07.926555    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:08.180093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.180053    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"]
Oct 30 07:24:08.180783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.180763    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f31ce3c0-821e-470f-9e38-b6c510825b81" containerName="revoke-keys"
Oct 30 07:24:08.180864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.180785    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31ce3c0-821e-470f-9e38-b6c510825b81" containerName="revoke-keys"
Oct 30 07:24:08.180913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.180907    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31ce3c0-821e-470f-9e38-b6c510825b81" containerName="revoke-keys"
Oct 30 07:24:08.184714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.184678    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.201845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.201806    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"]
Oct 30 07:24:08.281134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.281102    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:24:08.282107 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:08.282082    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaf1a25_d92c_4277_9fd6_9b80bccd95b0.slice/crio-7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a WatchSource:0}: Error finding container 7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a: Status 404 returned error can't find the container with id 7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a
Oct 30 07:24:08.284761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.284725    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:08.284849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.284798    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:08.284892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.284858    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:08.294961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.294922    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.295138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.294972    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsc6\" (UniqueName: \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.295138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.295110    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.396252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.396208    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.396498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.396261    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsc6\" (UniqueName: \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.396498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.396338    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.397428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.397399    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.398987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.398963    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.420586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.420554    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsc6\" (UniqueName: \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\") pod \"maas-api-revoke-keys-e2e-ipp-b-89664e-bght2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.497186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.496935    6738 patch_prober.go:29] interesting pod/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:24:08.497186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.497005    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:24:08.503173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.503143    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:08.504799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.504776    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"
Oct 30 07:24:08.504937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.504815    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f" event={"ID":"f31ce3c0-821e-470f-9e38-b6c510825b81","Type":"ContainerDied","Data":"b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a"}
Oct 30 07:24:08.504937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.504879    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73e5cf09a1314f2fa5c6eef1f34ab99a88b369cc0ef0320eff6fe5a86765f2a"
Oct 30 07:24:08.506780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.506754    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" event={"ID":"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0","Type":"ContainerStarted","Data":"91bc7948c650440d329e6dd41c0160744a67559898937337e7d736c19d3eb900"}
Oct 30 07:24:08.506919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.506806    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" event={"ID":"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0","Type":"ContainerStarted","Data":"7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a"}
Oct 30 07:24:08.558359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.558278    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podStartSLOduration=1.558258953 podStartE2EDuration="1.558258953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:08.556355822 +0000 UTC m=+1608.989119716" watchObservedRunningTime="2026-10-30 07:24:08.558258953 +0000 UTC m=+1608.991022845"
Oct 30 07:24:08.837382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.837347    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"]
Oct 30 07:24:08.838986 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:08.838958    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde435b40_d8e0_4d68_92dd_630712c3b4e2.slice/crio-6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51 WatchSource:0}: Error finding container 6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51: Status 404 returned error can't find the container with id 6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51
Oct 30 07:24:08.927578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:08.927536    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:09.408022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.407993    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.510964    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511012    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511053    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511212    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511227    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511239    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511721    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511836    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511878    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511919    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.511977    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\") pod \"c7979202-6d70-45ed-8ed5-8045828a4633\" (UID: \"c7979202-6d70-45ed-8ed5-8045828a4633\") "
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512212    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512308    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512461    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512607    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512621    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7979202-6d70-45ed-8ed5-8045828a4633-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512631    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512651    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.513935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.512662    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.515717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.515349    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "kube-api-access-pnbgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:09.515717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.515420    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:09.515717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.515579    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:09.516861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.516840    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token" pod "c7979202-6d70-45ed-8ed5-8045828a4633" (UID: "c7979202-6d70-45ed-8ed5-8045828a4633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:09.520541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.520509    6738 generic.go:356] "Generic (PLEG): container finished" podID="c7979202-6d70-45ed-8ed5-8045828a4633" containerID="f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b" exitCode=0
Oct 30 07:24:09.520642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.520559    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"
Oct 30 07:24:09.520642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.520583    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerDied","Data":"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"}
Oct 30 07:24:09.520768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.520640    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x" event={"ID":"c7979202-6d70-45ed-8ed5-8045828a4633","Type":"ContainerDied","Data":"0d8b474c89a7437c7d191ee24c029d38e8f68d2a159f795a00cacf4d0246ad9c"}
Oct 30 07:24:09.520768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.520671    6738 scope.go:122] "RemoveContainer" containerID="f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"
Oct 30 07:24:09.525714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.525607    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2" event={"ID":"de435b40-d8e0-4d68-92dd-630712c3b4e2","Type":"ContainerStarted","Data":"599b1822215d1cddbc26daaac08351d363f48d485aa71fce757c415805238229"}
Oct 30 07:24:09.525714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.525673    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2" event={"ID":"de435b40-d8e0-4d68-92dd-630712c3b4e2","Type":"ContainerStarted","Data":"6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51"}
Oct 30 07:24:09.537456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.537429    6738 scope.go:122] "RemoveContainer" containerID="a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"
Oct 30 07:24:09.549170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.549140    6738 scope.go:122] "RemoveContainer" containerID="f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"
Oct 30 07:24:09.549986 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:09.549949    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b\": container with ID starting with f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b not found: ID does not exist" containerID="f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"
Oct 30 07:24:09.550173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.550000    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b"} err="failed to get container status \"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b\": rpc error: code = NotFound desc = could not find container \"f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b\": container with ID starting with f58732620cfd07a2551741b499a3eba6cb3dc2eb66b977ae8d4b742cf69efc7b not found: ID does not exist"
Oct 30 07:24:09.550173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.550024    6738 scope.go:122] "RemoveContainer" containerID="a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"
Oct 30 07:24:09.550635 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:09.550469    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978\": container with ID starting with a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978 not found: ID does not exist" containerID="a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"
Oct 30 07:24:09.550635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.550504    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978"} err="failed to get container status \"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978\": rpc error: code = NotFound desc = could not find container \"a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978\": container with ID starting with a327f624d6722c24cd34e741ceb9b617ec2d6432a030e58190b385cb312aa978 not found: ID does not exist"
Oct 30 07:24:09.606884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.606841    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:24:09.609942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.609876    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-45b148-openshift-default-56dfdcdf-9dm2x"]
Oct 30 07:24:09.614988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.614584    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7979202-6d70-45ed-8ed5-8045828a4633-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.614988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.614614    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnbgq\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-kube-api-access-pnbgq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.614988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.614632    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7979202-6d70-45ed-8ed5-8045828a4633-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.614988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.614650    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7979202-6d70-45ed-8ed5-8045828a4633-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:09.629366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.628189    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:09.643898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.643844    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2" podStartSLOduration=1.6438291189999998 podStartE2EDuration="1.643829119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:09.642220025 +0000 UTC m=+1610.074983918" watchObservedRunningTime="2026-10-30 07:24:09.643829119 +0000 UTC m=+1610.076593011"
Oct 30 07:24:09.703020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.702988    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" path="/var/lib/kubelet/pods/c7979202-6d70-45ed-8ed5-8045828a4633/volumes"
Oct 30 07:24:09.929533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.929499    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:09.929744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:09.929551    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:10.303879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.303837    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:10.441333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.441282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:10.504069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.504001    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"]
Oct 30 07:24:10.522044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.522005    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2340b1-sxnw8"]
Oct 30 07:24:10.539237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.539208    6738 generic.go:356] "Generic (PLEG): container finished" podID="de435b40-d8e0-4d68-92dd-630712c3b4e2" containerID="599b1822215d1cddbc26daaac08351d363f48d485aa71fce757c415805238229" exitCode=0
Oct 30 07:24:10.539552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.539343    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2" event={"ID":"de435b40-d8e0-4d68-92dd-630712c3b4e2","Type":"ContainerDied","Data":"599b1822215d1cddbc26daaac08351d363f48d485aa71fce757c415805238229"}
Oct 30 07:24:10.644662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.644609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:10.928049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.928015    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:10.928241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.928068    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:10.979318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:10.978379    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" podUID="6ac77b74-ddb1-4bb1-84c2-962304650043" containerName="payload-processing" containerID="cri-o://f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd" gracePeriod=25
Oct 30 07:24:11.008268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.008224    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" podUID="6b7e5d4b-06c5-4b44-af55-352f0667354d" containerName="payload-pre-processing" containerID="cri-o://d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d" gracePeriod=25
Oct 30 07:24:11.029082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.029050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:11.344840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.344806    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:24:11.396332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.395326    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:24:11.448263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.448228    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\" (UniqueName: \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\") pod \"6b7e5d4b-06c5-4b44-af55-352f0667354d\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") "
Oct 30 07:24:11.448503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.448344    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\" (UniqueName: \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\") pod \"6ac77b74-ddb1-4bb1-84c2-962304650043\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") "
Oct 30 07:24:11.448503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.448431    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\") pod \"6ac77b74-ddb1-4bb1-84c2-962304650043\" (UID: \"6ac77b74-ddb1-4bb1-84c2-962304650043\") "
Oct 30 07:24:11.448503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.448474    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\") pod \"6b7e5d4b-06c5-4b44-af55-352f0667354d\" (UID: \"6b7e5d4b-06c5-4b44-af55-352f0667354d\") "
Oct 30 07:24:11.449349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.449317    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume" pod "6b7e5d4b-06c5-4b44-af55-352f0667354d" (UID: "6b7e5d4b-06c5-4b44-af55-352f0667354d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:11.450560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.450534    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume" pod "6ac77b74-ddb1-4bb1-84c2-962304650043" (UID: "6ac77b74-ddb1-4bb1-84c2-962304650043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:11.504098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.499665    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67" pod "6ac77b74-ddb1-4bb1-84c2-962304650043" (UID: "6ac77b74-ddb1-4bb1-84c2-962304650043"). InnerVolumeSpecName "kube-api-access-kww67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:11.504526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.504493    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n" pod "6b7e5d4b-06c5-4b44-af55-352f0667354d" (UID: "6b7e5d4b-06c5-4b44-af55-352f0667354d"). InnerVolumeSpecName "kube-api-access-tcf9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:11.550168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550138    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf9n\" (UniqueName: \"kubernetes.io/projected/6b7e5d4b-06c5-4b44-af55-352f0667354d-kube-api-access-tcf9n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.550168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550168    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kww67\" (UniqueName: \"kubernetes.io/projected/6ac77b74-ddb1-4bb1-84c2-962304650043-kube-api-access-kww67\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550182    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac77b74-ddb1-4bb1-84c2-962304650043-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550195    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e5d4b-06c5-4b44-af55-352f0667354d-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550319    6738 generic.go:356] "Generic (PLEG): container finished" podID="6ac77b74-ddb1-4bb1-84c2-962304650043" containerID="f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd" exitCode=0
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550382    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550405    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" event={"ID":"6ac77b74-ddb1-4bb1-84c2-962304650043","Type":"ContainerDied","Data":"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"}
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550488    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd" event={"ID":"6ac77b74-ddb1-4bb1-84c2-962304650043","Type":"ContainerDied","Data":"5e22c5612072c7ee4168b471154fd4112b49ded97f835c72d9ebae5f9a55e60d"}
Oct 30 07:24:11.550780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.550511    6738 scope.go:122] "RemoveContainer" containerID="f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"
Oct 30 07:24:11.552396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.552371    6738 generic.go:356] "Generic (PLEG): container finished" podID="6b7e5d4b-06c5-4b44-af55-352f0667354d" containerID="d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d" exitCode=0
Oct 30 07:24:11.553409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.552415    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"
Oct 30 07:24:11.553409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.552428    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" event={"ID":"6b7e5d4b-06c5-4b44-af55-352f0667354d","Type":"ContainerDied","Data":"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"}
Oct 30 07:24:11.553409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.552741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h" event={"ID":"6b7e5d4b-06c5-4b44-af55-352f0667354d","Type":"ContainerDied","Data":"61fc5880ea13bf56085a6570efba01a829a94f217705c3b31a582e8c74e83558"}
Oct 30 07:24:11.565264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.565242    6738 scope.go:122] "RemoveContainer" containerID="f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"
Oct 30 07:24:11.565665 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:11.565637    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd\": container with ID starting with f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd not found: ID does not exist" containerID="f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"
Oct 30 07:24:11.565767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.565678    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd"} err="failed to get container status \"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd\": rpc error: code = NotFound desc = could not find container \"f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd\": container with ID starting with f7e2bf297f8098188bf5acee556ead77d8788d436089cf51098b7215bbdbe1dd not found: ID does not exist"
Oct 30 07:24:11.565767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.565700    6738 scope.go:122] "RemoveContainer" containerID="d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"
Oct 30 07:24:11.609424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.609398    6738 scope.go:122] "RemoveContainer" containerID="d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"
Oct 30 07:24:11.610081 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:11.609845    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d\": container with ID starting with d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d not found: ID does not exist" containerID="d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"
Oct 30 07:24:11.610081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.609884    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d"} err="failed to get container status \"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d\": rpc error: code = NotFound desc = could not find container \"d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d\": container with ID starting with d8925c6a461dd17ad792bf8ae20ecf0aa42d380edcfd2bfebf2ee97ce662746d not found: ID does not exist"
Oct 30 07:24:11.616687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.616651    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"]
Oct 30 07:24:11.617395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617376    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6b7e5d4b-06c5-4b44-af55-352f0667354d" containerName="payload-pre-processing"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617396    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7e5d4b-06c5-4b44-af55-352f0667354d" containerName="payload-pre-processing"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617424    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617429    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617444    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617449    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617456    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6ac77b74-ddb1-4bb1-84c2-962304650043" containerName="payload-processing"
Oct 30 07:24:11.617460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617461    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac77b74-ddb1-4bb1-84c2-962304650043" containerName="payload-processing"
Oct 30 07:24:11.617693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617537    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617545    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7979202-6d70-45ed-8ed5-8045828a4633" containerName="istio-proxy"
Oct 30 07:24:11.617693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617552    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac77b74-ddb1-4bb1-84c2-962304650043" containerName="payload-processing"
Oct 30 07:24:11.617693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.617562    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7e5d4b-06c5-4b44-af55-352f0667354d" containerName="payload-pre-processing"
Oct 30 07:24:11.620748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.620728    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.638924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.638893    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"]
Oct 30 07:24:11.658185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.650175    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:24:11.658185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.654966    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2340b1-68fdd747d6-59zjd"]
Oct 30 07:24:11.713693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.712676    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c241b8-53f4-46cd-a9d7-47dd0729da47" path="/var/lib/kubelet/pods/64c241b8-53f4-46cd-a9d7-47dd0729da47/volumes"
Oct 30 07:24:11.714322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.713994    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac77b74-ddb1-4bb1-84c2-962304650043" path="/var/lib/kubelet/pods/6ac77b74-ddb1-4bb1-84c2-962304650043/volumes"
Oct 30 07:24:11.739936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.739905    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:24:11.747116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.747085    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2340b1-776b48697f-zvz4h"]
Oct 30 07:24:11.752409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.752378    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.752589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.752517    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kg8\" (UniqueName: \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.752589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.752562    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.805025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.804996    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:11.854004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.853972    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\") pod \"de435b40-d8e0-4d68-92dd-630712c3b4e2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") "
Oct 30 07:24:11.854640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.854620    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\") pod \"de435b40-d8e0-4d68-92dd-630712c3b4e2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") "
Oct 30 07:24:11.855140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.854554    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca" pod "de435b40-d8e0-4d68-92dd-630712c3b4e2" (UID: "de435b40-d8e0-4d68-92dd-630712c3b4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:11.855375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.855356    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\" (UniqueName: \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\") pod \"de435b40-d8e0-4d68-92dd-630712c3b4e2\" (UID: \"de435b40-d8e0-4d68-92dd-630712c3b4e2\") "
Oct 30 07:24:11.855864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.855847    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.856662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.856641    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kg8\" (UniqueName: \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.858195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858058    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.858395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858355    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6" pod "de435b40-d8e0-4d68-92dd-630712c3b4e2" (UID: "de435b40-d8e0-4d68-92dd-630712c3b4e2"). InnerVolumeSpecName "kube-api-access-zbsc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:11.858544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858524    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca" pod "de435b40-d8e0-4d68-92dd-630712c3b4e2" (UID: "de435b40-d8e0-4d68-92dd-630712c3b4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:11.858647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858632    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.858907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858894    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.858994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.858982    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de435b40-d8e0-4d68-92dd-630712c3b4e2-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.859073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.859063    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbsc6\" (UniqueName: \"kubernetes.io/projected/de435b40-d8e0-4d68-92dd-630712c3b4e2-kube-api-access-zbsc6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:11.863413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.859821    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.895256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.895156    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kg8\" (UniqueName: \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\") pod \"maas-api-revoke-sub-c0860cd5-d9cdj\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:11.927714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.927670    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:11.927896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.927733    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:11.935863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:11.935824    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:12.264246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.264224    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"]
Oct 30 07:24:12.266340 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:12.266270    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc706abed_1425_4ef9_bc1f_3d3afc005a70.slice/crio-6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c WatchSource:0}: Error finding container 6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c: Status 404 returned error can't find the container with id 6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c
Oct 30 07:24:12.562775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.561631    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2" event={"ID":"de435b40-d8e0-4d68-92dd-630712c3b4e2","Type":"ContainerDied","Data":"6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51"}
Oct 30 07:24:12.562775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.561690    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a3ab1a8d479918ec9aaaf543f50420bd532133ab7477b30eb2f5ec2b6156d51"
Oct 30 07:24:12.562775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.561751    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"
Oct 30 07:24:12.563629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.563599    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj" event={"ID":"c706abed-1425-4ef9-bc1f-3d3afc005a70","Type":"ContainerStarted","Data":"6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c"}
Oct 30 07:24:12.928320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.928107    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:12.928320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.928177    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:12.932031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:12.932005    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:24:13.032119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.032084    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"]
Oct 30 07:24:13.033530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.033498    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8238b38a-pvczv"]
Oct 30 07:24:13.102003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.101914    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.102259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.102243    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.102547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.102530    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.102981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.102962    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.103226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103212    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.103392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103377    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.103556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103541    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.103658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103645    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.103762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103577    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:13.103837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103815    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:13.103946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.103932    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\") pod \"d6a24618-7125-4e72-9da1-73900e657d6c\" (UID: \"d6a24618-7125-4e72-9da1-73900e657d6c\") "
Oct 30 07:24:13.104102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104081    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:13.104276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104256    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:13.104584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104039    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:13.104795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104779    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.104982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104962    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.105060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.104987    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.105060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.105001    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a24618-7125-4e72-9da1-73900e657d6c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.105060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.105016    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.106280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.106242    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:13.106476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.106274    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:13.106726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.106709    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:13.107673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.107650    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t" pod "d6a24618-7125-4e72-9da1-73900e657d6c" (UID: "d6a24618-7125-4e72-9da1-73900e657d6c"). InnerVolumeSpecName "kube-api-access-7zx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:13.206196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.206149    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zx9t\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-kube-api-access-7zx9t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.206196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.206186    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a24618-7125-4e72-9da1-73900e657d6c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.206196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.206199    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a24618-7125-4e72-9da1-73900e657d6c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.206513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.206211    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a24618-7125-4e72-9da1-73900e657d6c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:13.493446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.493188    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:13.599714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.599674    6738 generic.go:356] "Generic (PLEG): container finished" podID="d6a24618-7125-4e72-9da1-73900e657d6c" containerID="7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4" exitCode=0
Oct 30 07:24:13.600230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.599726    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"
Oct 30 07:24:13.600230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.599749    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerDied","Data":"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"}
Oct 30 07:24:13.600230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.599800    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78" event={"ID":"d6a24618-7125-4e72-9da1-73900e657d6c","Type":"ContainerDied","Data":"acc85dd18218d69594fee2cc03fbb7fbed582dc56bce4029c6bda8e7218c554e"}
Oct 30 07:24:13.600230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.599820    6738 scope.go:122] "RemoveContainer" containerID="7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"
Oct 30 07:24:13.602650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.602624    6738 generic.go:356] "Generic (PLEG): container finished" podID="c706abed-1425-4ef9-bc1f-3d3afc005a70" containerID="8d4d898473080dc7800bf1940213c15dab50a21e7f67860b6c120f77d2805134" exitCode=0
Oct 30 07:24:13.602849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.602704    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj" event={"ID":"c706abed-1425-4ef9-bc1f-3d3afc005a70","Type":"ContainerDied","Data":"8d4d898473080dc7800bf1940213c15dab50a21e7f67860b6c120f77d2805134"}
Oct 30 07:24:13.617989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.617963    6738 scope.go:122] "RemoveContainer" containerID="9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"
Oct 30 07:24:13.629863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.629518    6738 scope.go:122] "RemoveContainer" containerID="7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"
Oct 30 07:24:13.629950 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:13.629921    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4\": container with ID starting with 7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4 not found: ID does not exist" containerID="7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"
Oct 30 07:24:13.630014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.629956    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4"} err="failed to get container status \"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4\": rpc error: code = NotFound desc = could not find container \"7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4\": container with ID starting with 7a366771912eff44af95b207d6a6e1e898054e4bd66d95a6d1ecc569daf116b4 not found: ID does not exist"
Oct 30 07:24:13.630014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.629978    6738 scope.go:122] "RemoveContainer" containerID="9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"
Oct 30 07:24:13.630254 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:13.630234    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b\": container with ID starting with 9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b not found: ID does not exist" containerID="9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"
Oct 30 07:24:13.630361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.630258    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b"} err="failed to get container status \"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b\": rpc error: code = NotFound desc = could not find container \"9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b\": container with ID starting with 9e9dea80625a3994c66c2e35a01a895f7d9366b289bd31b6ad37b8c70b91c31b not found: ID does not exist"
Oct 30 07:24:13.653891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.653853    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:24:13.682332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.667569    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8d44bf-openshift-default-5fb6967b47-qcq78"]
Oct 30 07:24:13.699352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.699318    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7e5d4b-06c5-4b44-af55-352f0667354d" path="/var/lib/kubelet/pods/6b7e5d4b-06c5-4b44-af55-352f0667354d/volumes"
Oct 30 07:24:13.699788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.699775    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc8e5fb-9c51-466b-b4a0-3e681122f467" path="/var/lib/kubelet/pods/7dc8e5fb-9c51-466b-b4a0-3e681122f467/volumes"
Oct 30 07:24:13.700209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.700198    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" path="/var/lib/kubelet/pods/d6a24618-7125-4e72-9da1-73900e657d6c/volumes"
Oct 30 07:24:13.928515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.928428    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:13.928515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:13.928488    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:14.776565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.776540    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:14.927763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.927720    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\") pod \"c706abed-1425-4ef9-bc1f-3d3afc005a70\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") "
Oct 30 07:24:14.927763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.927747    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 07:24:14.928033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.927800    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 07:24:14.928033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.927824    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\" (UniqueName: \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\") pod \"c706abed-1425-4ef9-bc1f-3d3afc005a70\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") "
Oct 30 07:24:14.928033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.927987    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\") pod \"c706abed-1425-4ef9-bc1f-3d3afc005a70\" (UID: \"c706abed-1425-4ef9-bc1f-3d3afc005a70\") "
Oct 30 07:24:14.928478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.928447    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca" pod "c706abed-1425-4ef9-bc1f-3d3afc005a70" (UID: "c706abed-1425-4ef9-bc1f-3d3afc005a70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:14.930140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.930107    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8" pod "c706abed-1425-4ef9-bc1f-3d3afc005a70" (UID: "c706abed-1425-4ef9-bc1f-3d3afc005a70"). InnerVolumeSpecName "kube-api-access-l7kg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:14.930269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:14.930136    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca" pod "c706abed-1425-4ef9-bc1f-3d3afc005a70" (UID: "c706abed-1425-4ef9-bc1f-3d3afc005a70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:15.030013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.029962    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:15.030013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.029999    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kg8\" (UniqueName: \"kubernetes.io/projected/c706abed-1425-4ef9-bc1f-3d3afc005a70-kube-api-access-l7kg8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:15.030013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.030011    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c706abed-1425-4ef9-bc1f-3d3afc005a70-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:15.449806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.449761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:15.472516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.472471    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:24:15.504461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.504424    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:24:15.620244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.619916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:15.628710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.628674    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj" event={"ID":"c706abed-1425-4ef9-bc1f-3d3afc005a70","Type":"ContainerDied","Data":"6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c"}
Oct 30 07:24:15.628958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.628938    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6644707fbbfcdf2f9982dec76523ef5ff4042bfe627d475007850c021487283c"
Oct 30 07:24:15.629137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.629121    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"
Oct 30 07:24:15.749705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.749606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:15.791374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.791341    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"]
Oct 30 07:24:15.803256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.793210    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.804541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.804513    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.804721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.804709    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.805979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.805951    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.808145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.806746    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c706abed-1425-4ef9-bc1f-3d3afc005a70" containerName="revoke-keys"
Oct 30 07:24:15.808326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808276    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c706abed-1425-4ef9-bc1f-3d3afc005a70" containerName="revoke-keys"
Oct 30 07:24:15.808445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808429    6738 cpu_manager.go:414] "containerMap: removing container" podUID="de435b40-d8e0-4d68-92dd-630712c3b4e2" containerName="revoke-keys"
Oct 30 07:24:15.808521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808446    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de435b40-d8e0-4d68-92dd-630712c3b4e2" containerName="revoke-keys"
Oct 30 07:24:15.808719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808703    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c706abed-1425-4ef9-bc1f-3d3afc005a70" containerName="revoke-keys"
Oct 30 07:24:15.808791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808726    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="de435b40-d8e0-4d68-92dd-630712c3b4e2" containerName="revoke-keys"
Oct 30 07:24:15.808791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808746    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.808791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.808757    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a24618-7125-4e72-9da1-73900e657d6c" containerName="istio-proxy"
Oct 30 07:24:15.813202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.811656    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"]
Oct 30 07:24:15.813202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.811776    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:15.829915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.829882    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:15.830103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.830007    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:15.932879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.932837    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:15.933185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.933156    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:15.934038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.934020    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:24:15.941678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.941647    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5s4\" (UniqueName: \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:15.941824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.941697    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:15.941871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:15.941851    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.042918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.042838    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.043098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.042979    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.043192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.043173    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5s4\" (UniqueName: \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.044108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.044079    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.046348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.045657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.067393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.067360    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5s4\" (UniqueName: \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\") pod \"maas-api-revoke-sub-e9898b62-2cc7r\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.133659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.133623    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:16.537488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.537456    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"]
Oct 30 07:24:16.538379 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:16.538355    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0a2639_ceca_4525_8161_245aa10231e5.slice/crio-b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a WatchSource:0}: Error finding container b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a: Status 404 returned error can't find the container with id b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a
Oct 30 07:24:16.645121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:16.645083    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r" event={"ID":"cd0a2639-ceca-4525-8161-245aa10231e5","Type":"ContainerStarted","Data":"b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a"}
Oct 30 07:24:17.110882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.109023    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:24:17.110882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.109317    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" podUID="7a81f0c8-77b3-45ed-9084-b8464e21d512" containerName="maas-api" containerID="cri-o://a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415" gracePeriod=30
Oct 30 07:24:17.195582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.195541    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:17.506955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.506924    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:24:17.666340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.649226    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\" (UniqueName: \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\") pod \"7a81f0c8-77b3-45ed-9084-b8464e21d512\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") "
Oct 30 07:24:17.666340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.649327    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\") pod \"7a81f0c8-77b3-45ed-9084-b8464e21d512\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") "
Oct 30 07:24:17.666340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.649434    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\") pod \"7a81f0c8-77b3-45ed-9084-b8464e21d512\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") "
Oct 30 07:24:17.666340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.649561    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\") pod \"7a81f0c8-77b3-45ed-9084-b8464e21d512\" (UID: \"7a81f0c8-77b3-45ed-9084-b8464e21d512\") "
Oct 30 07:24:17.666340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.660554    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca" pod "7a81f0c8-77b3-45ed-9084-b8464e21d512" (UID: "7a81f0c8-77b3-45ed-9084-b8464e21d512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:17.693149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.687517    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk" pod "7a81f0c8-77b3-45ed-9084-b8464e21d512" (UID: "7a81f0c8-77b3-45ed-9084-b8464e21d512"). InnerVolumeSpecName "kube-api-access-k66vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:17.700391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.700330    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs" pod "7a81f0c8-77b3-45ed-9084-b8464e21d512" (UID: "7a81f0c8-77b3-45ed-9084-b8464e21d512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:17.700747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.700709    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls" pod "7a81f0c8-77b3-45ed-9084-b8464e21d512" (UID: "7a81f0c8-77b3-45ed-9084-b8464e21d512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:17.751851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.748429    6738 generic.go:356] "Generic (PLEG): container finished" podID="7a81f0c8-77b3-45ed-9084-b8464e21d512" containerID="a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415" exitCode=0
Oct 30 07:24:17.751851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.748682    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"
Oct 30 07:24:17.751851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.749729    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" event={"ID":"7a81f0c8-77b3-45ed-9084-b8464e21d512","Type":"ContainerDied","Data":"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"}
Oct 30 07:24:17.751851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.749778    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws" event={"ID":"7a81f0c8-77b3-45ed-9084-b8464e21d512","Type":"ContainerDied","Data":"5880e20cd27dbc05c47f813e0dee376096da607861631b1be86cc49e5d277d6a"}
Oct 30 07:24:17.751851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.749805    6738 scope.go:122] "RemoveContainer" containerID="a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"
Oct 30 07:24:17.760538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.760512    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:17.761365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.761351    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a81f0c8-77b3-45ed-9084-b8464e21d512-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:17.761491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.761476    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k66vk\" (UniqueName: \"kubernetes.io/projected/7a81f0c8-77b3-45ed-9084-b8464e21d512-kube-api-access-k66vk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:17.761585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.761574    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a81f0c8-77b3-45ed-9084-b8464e21d512-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:17.761665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.761115    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r" event={"ID":"cd0a2639-ceca-4525-8161-245aa10231e5","Type":"ContainerStarted","Data":"0bd5644d83e5aaaf6283c3418ab91ef594eb1b367807f7b05e3ca74d9bb3e3e6"}
Oct 30 07:24:17.778952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.778924    6738 scope.go:122] "RemoveContainer" containerID="a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"
Oct 30 07:24:17.779376 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:17.779346    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415\": container with ID starting with a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415 not found: ID does not exist" containerID="a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"
Oct 30 07:24:17.779512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.779386    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415"} err="failed to get container status \"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415\": rpc error: code = NotFound desc = could not find container \"a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415\": container with ID starting with a0c34af25b992f9c8e720bdd08d3ddb1be0bd20172f40b7b9518e171903cf415 not found: ID does not exist"
Oct 30 07:24:17.833324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.833256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:17.853320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.853240    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r" podStartSLOduration=2.853218385 podStartE2EDuration="2.853218385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:17.813474156 +0000 UTC m=+1618.246238049" watchObservedRunningTime="2026-10-30 07:24:17.853218385 +0000 UTC m=+1618.285982280"
Oct 30 07:24:17.853554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.853532    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:24:17.867088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.867053    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-93de68-7598bd97fb-fplws"]
Oct 30 07:24:17.945963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:17.945909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:18.072977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.072939    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"]
Oct 30 07:24:18.092346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.089650    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:18.109094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.109052    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93de68-rbm2f"]
Oct 30 07:24:18.216318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.210576    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:24:18.216318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.211215    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" containerID="cri-o://13c5e003e549a5147986e76a783d04758eb1359ac5dbe63dc6c5a4588fa7efc6" gracePeriod=30
Oct 30 07:24:18.352800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.352703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:18.555838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.555793    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:18.771382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.771334    6738 generic.go:356] "Generic (PLEG): container finished" podID="cd0a2639-ceca-4525-8161-245aa10231e5" containerID="0bd5644d83e5aaaf6283c3418ab91ef594eb1b367807f7b05e3ca74d9bb3e3e6" exitCode=0
Oct 30 07:24:18.771912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:18.771880    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r" event={"ID":"cd0a2639-ceca-4525-8161-245aa10231e5","Type":"ContainerDied","Data":"0bd5644d83e5aaaf6283c3418ab91ef594eb1b367807f7b05e3ca74d9bb3e3e6"}
Oct 30 07:24:19.689569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.689539    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a81f0c8-77b3-45ed-9084-b8464e21d512" path="/var/lib/kubelet/pods/7a81f0c8-77b3-45ed-9084-b8464e21d512/volumes"
Oct 30 07:24:19.689998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.689983    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31ce3c0-821e-470f-9e38-b6c510825b81" path="/var/lib/kubelet/pods/f31ce3c0-821e-470f-9e38-b6c510825b81/volumes"
Oct 30 07:24:19.946362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.946282    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:19.971658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.971622    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\") pod \"cd0a2639-ceca-4525-8161-245aa10231e5\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") "
Oct 30 07:24:19.971865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.971689    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\" (UniqueName: \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\") pod \"cd0a2639-ceca-4525-8161-245aa10231e5\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") "
Oct 30 07:24:19.971865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.971734    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\") pod \"cd0a2639-ceca-4525-8161-245aa10231e5\" (UID: \"cd0a2639-ceca-4525-8161-245aa10231e5\") "
Oct 30 07:24:19.972064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.972040    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca" pod "cd0a2639-ceca-4525-8161-245aa10231e5" (UID: "cd0a2639-ceca-4525-8161-245aa10231e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:19.972224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.972205    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:19.974065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.974034    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4" pod "cd0a2639-ceca-4525-8161-245aa10231e5" (UID: "cd0a2639-ceca-4525-8161-245aa10231e5"). InnerVolumeSpecName "kube-api-access-dr5s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:19.974202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:19.974082    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca" pod "cd0a2639-ceca-4525-8161-245aa10231e5" (UID: "cd0a2639-ceca-4525-8161-245aa10231e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:20.073197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.073153    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr5s4\" (UniqueName: \"kubernetes.io/projected/cd0a2639-ceca-4525-8161-245aa10231e5-kube-api-access-dr5s4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:20.073197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.073189    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a2639-ceca-4525-8161-245aa10231e5-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:20.255885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.255776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:20.472888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.472840    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" podUID="ca713360-0759-445e-88fa-5bf0e0014adb" containerName="payload-processing" containerID="cri-o://89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6" gracePeriod=25
Oct 30 07:24:20.505314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.505257    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" podUID="773eebc8-9a79-44c4-a833-1afc81e138ef" containerName="payload-pre-processing" containerID="cri-o://84a5e860e3fbb22db61c41fc58c30ddf1a590809c7f24fe7d81f4d59d39a69c8" gracePeriod=25
Oct 30 07:24:20.782721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.782689    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:24:20.792944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.792859    6738 generic.go:356] "Generic (PLEG): container finished" podID="ca713360-0759-445e-88fa-5bf0e0014adb" containerID="89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6" exitCode=0
Oct 30 07:24:20.792944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.792912    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"
Oct 30 07:24:20.792944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.792932    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" event={"ID":"ca713360-0759-445e-88fa-5bf0e0014adb","Type":"ContainerDied","Data":"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"}
Oct 30 07:24:20.793229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.792988    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv" event={"ID":"ca713360-0759-445e-88fa-5bf0e0014adb","Type":"ContainerDied","Data":"7cceff3d753b23a91f7efbee4b4cafbdf407fab75f2f41a55f4472d0e74a5430"}
Oct 30 07:24:20.793229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.793011    6738 scope.go:122] "RemoveContainer" containerID="89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"
Oct 30 07:24:20.795179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.795155    6738 generic.go:356] "Generic (PLEG): container finished" podID="773eebc8-9a79-44c4-a833-1afc81e138ef" containerID="84a5e860e3fbb22db61c41fc58c30ddf1a590809c7f24fe7d81f4d59d39a69c8" exitCode=0
Oct 30 07:24:20.795289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.795224    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" event={"ID":"773eebc8-9a79-44c4-a833-1afc81e138ef","Type":"ContainerDied","Data":"84a5e860e3fbb22db61c41fc58c30ddf1a590809c7f24fe7d81f4d59d39a69c8"}
Oct 30 07:24:20.797116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.797099    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"
Oct 30 07:24:20.797243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.797114    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r" event={"ID":"cd0a2639-ceca-4525-8161-245aa10231e5","Type":"ContainerDied","Data":"b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a"}
Oct 30 07:24:20.797243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.797157    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51ee06afed0c6237432bb6e1454a0ec7b122e1bfadc814b1fdd174e6dd1871a"
Oct 30 07:24:20.805992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.805956    6738 scope.go:122] "RemoveContainer" containerID="89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"
Oct 30 07:24:20.806350 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:20.806288    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6\": container with ID starting with 89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6 not found: ID does not exist" containerID="89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"
Oct 30 07:24:20.806434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.806360    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6"} err="failed to get container status \"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6\": rpc error: code = NotFound desc = could not find container \"89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6\": container with ID starting with 89a8ee0ee20fd38c4353b278713679105f6b5a067df47574695885102526f5a6 not found: ID does not exist"
Oct 30 07:24:20.806498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.806482    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:24:20.883392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.883354    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\" (UniqueName: \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\") pod \"ca713360-0759-445e-88fa-5bf0e0014adb\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") "
Oct 30 07:24:20.883646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.883443    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\") pod \"ca713360-0759-445e-88fa-5bf0e0014adb\" (UID: \"ca713360-0759-445e-88fa-5bf0e0014adb\") "
Oct 30 07:24:20.883646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.883495    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\" (UniqueName: \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\") pod \"773eebc8-9a79-44c4-a833-1afc81e138ef\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") "
Oct 30 07:24:20.883646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.883613    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\") pod \"773eebc8-9a79-44c4-a833-1afc81e138ef\" (UID: \"773eebc8-9a79-44c4-a833-1afc81e138ef\") "
Oct 30 07:24:20.884057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.884031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume" pod "ca713360-0759-445e-88fa-5bf0e0014adb" (UID: "ca713360-0759-445e-88fa-5bf0e0014adb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:20.884190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.884175    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca713360-0759-445e-88fa-5bf0e0014adb-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:20.884449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.884425    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume" pod "773eebc8-9a79-44c4-a833-1afc81e138ef" (UID: "773eebc8-9a79-44c4-a833-1afc81e138ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:20.886727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.886695    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g" pod "773eebc8-9a79-44c4-a833-1afc81e138ef" (UID: "773eebc8-9a79-44c4-a833-1afc81e138ef"). InnerVolumeSpecName "kube-api-access-vjc9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:20.886843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.886799    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6" pod "ca713360-0759-445e-88fa-5bf0e0014adb" (UID: "ca713360-0759-445e-88fa-5bf0e0014adb"). InnerVolumeSpecName "kube-api-access-j9cq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:20.986089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.986048    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773eebc8-9a79-44c4-a833-1afc81e138ef-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:20.986089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.986085    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9cq6\" (UniqueName: \"kubernetes.io/projected/ca713360-0759-445e-88fa-5bf0e0014adb-kube-api-access-j9cq6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:20.986089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:20.986097    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjc9g\" (UniqueName: \"kubernetes.io/projected/773eebc8-9a79-44c4-a833-1afc81e138ef-kube-api-access-vjc9g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:21.139666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.139635    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:24:21.143141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.143111    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93de68-869c596d8d-mjllv"]
Oct 30 07:24:21.690179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.690146    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca713360-0759-445e-88fa-5bf0e0014adb" path="/var/lib/kubelet/pods/ca713360-0759-445e-88fa-5bf0e0014adb/volumes"
Oct 30 07:24:21.807176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.807141    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"
Oct 30 07:24:21.807176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.807165    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs" event={"ID":"773eebc8-9a79-44c4-a833-1afc81e138ef","Type":"ContainerDied","Data":"5de715774fcd250c129577ab1df3c1dcf3de2a022e4e93d238b77fbe4387dd70"}
Oct 30 07:24:21.807718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.807215    6738 scope.go:122] "RemoveContainer" containerID="84a5e860e3fbb22db61c41fc58c30ddf1a590809c7f24fe7d81f4d59d39a69c8"
Oct 30 07:24:21.862612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.862579    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:24:21.874734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:21.874699    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93de68-6f97ddcd95-ll2vs"]
Oct 30 07:24:22.029769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.029690    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"]
Oct 30 07:24:22.030368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030345    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ca713360-0759-445e-88fa-5bf0e0014adb" containerName="payload-processing"
Oct 30 07:24:22.030368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030365    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca713360-0759-445e-88fa-5bf0e0014adb" containerName="payload-processing"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030400    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cd0a2639-ceca-4525-8161-245aa10231e5" containerName="revoke-keys"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030406    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0a2639-ceca-4525-8161-245aa10231e5" containerName="revoke-keys"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030421    6738 cpu_manager.go:414] "containerMap: removing container" podUID="773eebc8-9a79-44c4-a833-1afc81e138ef" containerName="payload-pre-processing"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030427    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773eebc8-9a79-44c4-a833-1afc81e138ef" containerName="payload-pre-processing"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030442    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7a81f0c8-77b3-45ed-9084-b8464e21d512" containerName="maas-api"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030447    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a81f0c8-77b3-45ed-9084-b8464e21d512" containerName="maas-api"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030521    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0a2639-ceca-4525-8161-245aa10231e5" containerName="revoke-keys"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030530    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="773eebc8-9a79-44c4-a833-1afc81e138ef" containerName="payload-pre-processing"
Oct 30 07:24:22.030536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030539    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a81f0c8-77b3-45ed-9084-b8464e21d512" containerName="maas-api"
Oct 30 07:24:22.030860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.030547    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca713360-0759-445e-88fa-5bf0e0014adb" containerName="payload-processing"
Oct 30 07:24:22.033937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.033918    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.049618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.049586    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:24:22.049618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.049598    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:22.049845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.049620    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:24:22.049996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.049977    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:22.054119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.054091    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"]
Oct 30 07:24:22.098703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.098670    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk25v\" (UniqueName: \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.098896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.098723    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.098896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.098838    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.188639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.188595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:22.199961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.199925    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk25v\" (UniqueName: \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.200138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.199978    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.200138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.200035    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.200763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.200740    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.202504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.202478    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.225111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.225077    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk25v\" (UniqueName: \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\") pod \"maas-api-revoke-sub-bf4d3e92-mb7ct\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.349161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.349063    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:22.652416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.652389    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"]
Oct 30 07:24:22.653157 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:22.653126    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddaaf8c_07ce_490e_9fe2_47c255578dd7.slice/crio-3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3 WatchSource:0}: Error finding container 3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3: Status 404 returned error can't find the container with id 3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3
Oct 30 07:24:22.815711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:22.815667    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct" event={"ID":"cddaaf8c-07ce-490e-9fe2-47c255578dd7","Type":"ContainerStarted","Data":"3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3"}
Oct 30 07:24:23.043074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.042987    6738 patch_prober.go:29] interesting pod/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:24:23.043074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.043047    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:24:23.722326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.719951    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773eebc8-9a79-44c4-a833-1afc81e138ef" path="/var/lib/kubelet/pods/773eebc8-9a79-44c4-a833-1afc81e138ef/volumes"
Oct 30 07:24:23.832787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.832696    6738 generic.go:356] "Generic (PLEG): container finished" podID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerID="13c5e003e549a5147986e76a783d04758eb1359ac5dbe63dc6c5a4588fa7efc6" exitCode=0
Oct 30 07:24:23.833335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.832792    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerDied","Data":"13c5e003e549a5147986e76a783d04758eb1359ac5dbe63dc6c5a4588fa7efc6"}
Oct 30 07:24:23.833335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.832879    6738 scope.go:122] "RemoveContainer" containerID="fab15fbcee5b3b59fef0ff4b9949243cb21c6ebdf5b3c075e8197f774d2ca551"
Oct 30 07:24:23.836651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.836620    6738 generic.go:356] "Generic (PLEG): container finished" podID="cddaaf8c-07ce-490e-9fe2-47c255578dd7" containerID="96832e11b0f564d78e93578866acc6643eee8638b3d2f5b8a2f4ff5b558e9e17" exitCode=0
Oct 30 07:24:23.836764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.836682    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct" event={"ID":"cddaaf8c-07ce-490e-9fe2-47c255578dd7","Type":"ContainerDied","Data":"96832e11b0f564d78e93578866acc6643eee8638b3d2f5b8a2f4ff5b558e9e17"}
Oct 30 07:24:23.946173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:23.946147    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:24:24.032944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.032900    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.032975    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033052    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033134    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033211    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033319    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033333    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:24.033395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033377    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033413    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033500    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\") pod \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\" (UID: \"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58\") "
Oct 30 07:24:24.033747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033633    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:24.033853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033832    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:24.033919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033843    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:24.033983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.033966    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:24.034230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.034141    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.034230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.034165    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.034230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.034182    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.034230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.034195    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.034230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.034207    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.035959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.035908    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:24.036081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.036043    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:24.038163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.038110    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:24.038272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.038202    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd" pod "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" (UID: "bbfdc2bf-8d8c-4396-bf35-43a515a5fd58"). InnerVolumeSpecName "kube-api-access-zbdjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:24.135513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.135482    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.135513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.135512    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.135513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.135521    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.135845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.135529    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbdjd\" (UniqueName: \"kubernetes.io/projected/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58-kube-api-access-zbdjd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:24.169282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.169245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:24.325518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.325477    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:24.448728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.448634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:24.645731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.645699    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:24:24.646344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.645962    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy" containerID="cri-o://5c91fe7b706116968dba21268be4f4662c83544381bb80a865aa0a8b283f2821" gracePeriod=30
Oct 30 07:24:24.683733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.683700    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:24.849735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.849656    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"
Oct 30 07:24:24.849735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.849694    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx" event={"ID":"bbfdc2bf-8d8c-4396-bf35-43a515a5fd58","Type":"ContainerDied","Data":"769fc8eafa589d8f255ccef4f333475d80f44b8b71d5806e946c2f6575fda3d5"}
Oct 30 07:24:24.850243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.849750    6738 scope.go:122] "RemoveContainer" containerID="13c5e003e549a5147986e76a783d04758eb1359ac5dbe63dc6c5a4588fa7efc6"
Oct 30 07:24:24.961280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.961242    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:24:24.999955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:24.999922    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2340b1-openshift-default-5684f9d4d9-4wwhx"]
Oct 30 07:24:25.008890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.008864    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:25.045780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.045748    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\") pod \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") "
Oct 30 07:24:25.045961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.045800    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\") pod \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") "
Oct 30 07:24:25.045961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.045904    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\" (UniqueName: \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\") pod \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\" (UID: \"cddaaf8c-07ce-490e-9fe2-47c255578dd7\") "
Oct 30 07:24:25.046128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.046109    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca" pod "cddaaf8c-07ce-490e-9fe2-47c255578dd7" (UID: "cddaaf8c-07ce-490e-9fe2-47c255578dd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:25.046365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.046343    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:25.047985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.047962    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca" pod "cddaaf8c-07ce-490e-9fe2-47c255578dd7" (UID: "cddaaf8c-07ce-490e-9fe2-47c255578dd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:25.048102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.047996    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v" pod "cddaaf8c-07ce-490e-9fe2-47c255578dd7" (UID: "cddaaf8c-07ce-490e-9fe2-47c255578dd7"). InnerVolumeSpecName "kube-api-access-zk25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:25.147313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.147257    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddaaf8c-07ce-490e-9fe2-47c255578dd7-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:25.147498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.147334    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk25v\" (UniqueName: \"kubernetes.io/projected/cddaaf8c-07ce-490e-9fe2-47c255578dd7-kube-api-access-zk25v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:25.689545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.689513    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" path="/var/lib/kubelet/pods/bbfdc2bf-8d8c-4396-bf35-43a515a5fd58/volumes"
Oct 30 07:24:25.862364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.862333    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"
Oct 30 07:24:25.863386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.862388    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct" event={"ID":"cddaaf8c-07ce-490e-9fe2-47c255578dd7","Type":"ContainerDied","Data":"3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3"}
Oct 30 07:24:25.864044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.864022    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc693ad4f029e5bf76442654f30ec3ec2f1004216cc5fb7653bef1548cfb3c3"
Oct 30 07:24:25.894588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.894553    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:25.994523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.994440    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:24:25.994686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:25.994646    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" podUID="56b2b062-3098-48b5-b850-560c8fdaac08" containerName="payload-processing" containerID="cri-o://a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279" gracePeriod=30
Oct 30 07:24:26.016358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.016325    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:24:26.016590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.016564    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" podUID="10baaa70-ea0c-482a-9e67-968a8573954e" containerName="payload-pre-processing" containerID="cri-o://4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e" gracePeriod=30
Oct 30 07:24:26.089350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.089309    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:26.201337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.200355    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:26.304681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.304653    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:24:26.369361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.362402    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\") pod \"56b2b062-3098-48b5-b850-560c8fdaac08\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") "
Oct 30 07:24:26.369361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.362478    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\" (UniqueName: \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\") pod \"56b2b062-3098-48b5-b850-560c8fdaac08\" (UID: \"56b2b062-3098-48b5-b850-560c8fdaac08\") "
Oct 30 07:24:26.406813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.406758    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume" pod "56b2b062-3098-48b5-b850-560c8fdaac08" (UID: "56b2b062-3098-48b5-b850-560c8fdaac08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:26.418674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.414912    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk" pod "56b2b062-3098-48b5-b850-560c8fdaac08" (UID: "56b2b062-3098-48b5-b850-560c8fdaac08"). InnerVolumeSpecName "kube-api-access-9whpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:26.424315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.424072    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:24:26.464076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.464042    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\" (UniqueName: \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\") pod \"10baaa70-ea0c-482a-9e67-968a8573954e\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") "
Oct 30 07:24:26.464243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.464140    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\") pod \"10baaa70-ea0c-482a-9e67-968a8573954e\" (UID: \"10baaa70-ea0c-482a-9e67-968a8573954e\") "
Oct 30 07:24:26.464470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.464454    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2b062-3098-48b5-b850-560c8fdaac08-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:26.464555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.464471    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9whpk\" (UniqueName: \"kubernetes.io/projected/56b2b062-3098-48b5-b850-560c8fdaac08-kube-api-access-9whpk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:26.464555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.464525    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume" pod "10baaa70-ea0c-482a-9e67-968a8573954e" (UID: "10baaa70-ea0c-482a-9e67-968a8573954e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:26.467573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.467539    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l" pod "10baaa70-ea0c-482a-9e67-968a8573954e" (UID: "10baaa70-ea0c-482a-9e67-968a8573954e"). InnerVolumeSpecName "kube-api-access-pj48l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:26.565773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.565681    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj48l\" (UniqueName: \"kubernetes.io/projected/10baaa70-ea0c-482a-9e67-968a8573954e-kube-api-access-pj48l\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:26.565773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.565718    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10baaa70-ea0c-482a-9e67-968a8573954e-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:26.599050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.599010    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:26.717452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.717417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:26.742594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.742561    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"]
Oct 30 07:24:26.746771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.746741    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89664e-bght2"]
Oct 30 07:24:26.839434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.838907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:26.874112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.874079    6738 generic.go:356] "Generic (PLEG): container finished" podID="10baaa70-ea0c-482a-9e67-968a8573954e" containerID="4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e" exitCode=0
Oct 30 07:24:26.874678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.874148    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"
Oct 30 07:24:26.874678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.874171    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" event={"ID":"10baaa70-ea0c-482a-9e67-968a8573954e","Type":"ContainerDied","Data":"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"}
Oct 30 07:24:26.874678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.874228    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg" event={"ID":"10baaa70-ea0c-482a-9e67-968a8573954e","Type":"ContainerDied","Data":"787d00fd2a1ca82e9249a2e51091a71929db0c9721b015e5bda16b1a8ac3451b"}
Oct 30 07:24:26.874678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.874263    6738 scope.go:122] "RemoveContainer" containerID="4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"
Oct 30 07:24:26.875861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.875813    6738 generic.go:356] "Generic (PLEG): container finished" podID="56b2b062-3098-48b5-b850-560c8fdaac08" containerID="a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279" exitCode=0
Oct 30 07:24:26.875922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.875859    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" event={"ID":"56b2b062-3098-48b5-b850-560c8fdaac08","Type":"ContainerDied","Data":"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"}
Oct 30 07:24:26.875922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.875910    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk" event={"ID":"56b2b062-3098-48b5-b850-560c8fdaac08","Type":"ContainerDied","Data":"7eb5c5c93ad5ecdb37566a9b0282dc60d5ee56b58ea17bc197c51b32a055ce9c"}
Oct 30 07:24:26.876026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.875990    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"
Oct 30 07:24:26.888419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.888380    6738 scope.go:122] "RemoveContainer" containerID="4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"
Oct 30 07:24:26.888419 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:26.889616    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e\": container with ID starting with 4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e not found: ID does not exist" containerID="4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"
Oct 30 07:24:26.888419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.889657    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e"} err="failed to get container status \"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e\": rpc error: code = NotFound desc = could not find container \"4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e\": container with ID starting with 4c71beb5772d6320f6143c4d893412dcc6d53f5204f8d32a00e30e3211baef2e not found: ID does not exist"
Oct 30 07:24:26.888419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.889685    6738 scope.go:122] "RemoveContainer" containerID="a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"
Oct 30 07:24:26.906730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.906705    6738 scope.go:122] "RemoveContainer" containerID="a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"
Oct 30 07:24:26.907047 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:26.907026    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279\": container with ID starting with a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279 not found: ID does not exist" containerID="a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"
Oct 30 07:24:26.907097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.907057    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279"} err="failed to get container status \"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279\": rpc error: code = NotFound desc = could not find container \"a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279\": container with ID starting with a2311c59dbad8811d19f560d902bc26cf566b776c9c5834a16cb2d0a618fc279 not found: ID does not exist"
Oct 30 07:24:26.942345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.942310    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:24:26.945902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.945874    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89664e-57cf67946c-sjmpg"]
Oct 30 07:24:26.980003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.979967    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:24:26.984191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:26.984155    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89664e-8f7dd45b8-dv2xk"]
Oct 30 07:24:27.439741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:27.439703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:27.689328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:27.689278    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10baaa70-ea0c-482a-9e67-968a8573954e" path="/var/lib/kubelet/pods/10baaa70-ea0c-482a-9e67-968a8573954e/volumes"
Oct 30 07:24:27.689725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:27.689712    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b2b062-3098-48b5-b850-560c8fdaac08" path="/var/lib/kubelet/pods/56b2b062-3098-48b5-b850-560c8fdaac08/volumes"
Oct 30 07:24:27.690129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:27.690091    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de435b40-d8e0-4d68-92dd-630712c3b4e2" path="/var/lib/kubelet/pods/de435b40-d8e0-4d68-92dd-630712c3b4e2/volumes"
Oct 30 07:24:28.948065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:28.948026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:29.901522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.901483    6738 generic.go:356] "Generic (PLEG): container finished" podID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerID="5c91fe7b706116968dba21268be4f4662c83544381bb80a865aa0a8b283f2821" exitCode=0
Oct 30 07:24:29.901681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.901538    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerDied","Data":"5c91fe7b706116968dba21268be4f4662c83544381bb80a865aa0a8b283f2821"}
Oct 30 07:24:29.901681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.901599    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7" event={"ID":"a994cc50-7ea6-486e-9ad7-5bc0094d54b1","Type":"ContainerDied","Data":"473c7cb79c5b33dc29008dc017c3d1582d37b659938fb23b1a676f35dbcf836c"}
Oct 30 07:24:29.901681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.901619    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473c7cb79c5b33dc29008dc017c3d1582d37b659938fb23b1a676f35dbcf836c"
Oct 30 07:24:29.901681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.901633    6738 scope.go:122] "RemoveContainer" containerID="4fa4f12c860e46186b0d8c037b25b425513e020758b674d81c424e948cd2f7ac"
Oct 30 07:24:29.917037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:29.917014    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:24:30.018843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.018799    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.018928    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019043    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019095    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019134    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019181    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019231    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019263    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019336    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\") pod \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\" (UID: \"a994cc50-7ea6-486e-9ad7-5bc0094d54b1\") "
Oct 30 07:24:30.019723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019424    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:30.019723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019434    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.019723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019537    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.019723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019550    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019762    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019920    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019939    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019954    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019968    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.019986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.019980    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.021400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.021372    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.021510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.021483    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.021725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.021705    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "kube-api-access-jrnlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.021855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.021832    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo" pod "a994cc50-7ea6-486e-9ad7-5bc0094d54b1" (UID: "a994cc50-7ea6-486e-9ad7-5bc0094d54b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:30.120889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.120853    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.120889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.120881    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.120889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.120891    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.121135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.120904    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnlh\" (UniqueName: \"kubernetes.io/projected/a994cc50-7ea6-486e-9ad7-5bc0094d54b1-kube-api-access-jrnlh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.427425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.427382    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:30.811556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.811444    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:30.910879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:30.910849    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"
Oct 30 07:24:31.062413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:31.062321    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:24:31.067043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:31.067005    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-93de68-openshift-default-5fcd95fd8f-5wnr7"]
Oct 30 07:24:31.689790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:31.689731    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" path="/var/lib/kubelet/pods/a994cc50-7ea6-486e-9ad7-5bc0094d54b1/volumes"
Oct 30 07:24:32.765695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:32.765645    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:33.151563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.151523    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"]
Oct 30 07:24:33.152083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152055    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:33.152083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152079    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152102    6738 cpu_manager.go:414] "containerMap: removing container" podUID="56b2b062-3098-48b5-b850-560c8fdaac08" containerName="payload-processing"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152110    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b2b062-3098-48b5-b850-560c8fdaac08" containerName="payload-processing"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152121    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cddaaf8c-07ce-490e-9fe2-47c255578dd7" containerName="revoke-keys"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152130    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddaaf8c-07ce-490e-9fe2-47c255578dd7" containerName="revoke-keys"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152146    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152152    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152166    6738 cpu_manager.go:414] "containerMap: removing container" podUID="10baaa70-ea0c-482a-9e67-968a8573954e" containerName="payload-pre-processing"
Oct 30 07:24:33.152186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152171    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10baaa70-ea0c-482a-9e67-968a8573954e" containerName="payload-pre-processing"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152288    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152319    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b2b062-3098-48b5-b850-560c8fdaac08" containerName="payload-processing"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152327    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddaaf8c-07ce-490e-9fe2-47c255578dd7" containerName="revoke-keys"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152333    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152339    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="10baaa70-ea0c-482a-9e67-968a8573954e" containerName="payload-pre-processing"
Oct 30 07:24:33.152442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.152346    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:33.156869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.156841    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.172631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.172604    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"]
Oct 30 07:24:33.182378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.182349    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:33.182378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.182350    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:24:33.182566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.182360    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:24:33.187902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.187872    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:33.273473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.273430    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.273682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.273493    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmn7\" (UniqueName: \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.273682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.273596    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.375266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.375227    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.375266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.375280    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmn7\" (UniqueName: \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.375521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.375356    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.375919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.375892    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.377858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.377834    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.399254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.399225    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmn7\" (UniqueName: \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\") pod \"maas-api-revoke-sub-65d76536-24n8g\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.472174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.472077    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:33.786697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.786664    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"]
Oct 30 07:24:33.787162 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:33.787136    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2349be3_5aea_4165_8fa7_afd82b46caf4.slice/crio-a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b WatchSource:0}: Error finding container a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b: Status 404 returned error can't find the container with id a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b
Oct 30 07:24:33.934031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:33.933993    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g" event={"ID":"e2349be3-5aea-4165-8fa7-afd82b46caf4","Type":"ContainerStarted","Data":"a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b"}
Oct 30 07:24:34.002246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.002206    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:34.151897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.151856    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:34.344553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.344456    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:24:34.344743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.344717    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy" containerID="cri-o://9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151" gracePeriod=30
Oct 30 07:24:34.397768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.397730    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"]
Oct 30 07:24:34.398423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.398400    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:34.398423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.398421    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:34.398646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.398445    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:34.398646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.398454    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfdc2bf-8d8c-4396-bf35-43a515a5fd58" containerName="istio-proxy"
Oct 30 07:24:34.398646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.398578    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a994cc50-7ea6-486e-9ad7-5bc0094d54b1" containerName="istio-proxy"
Oct 30 07:24:34.402079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.402058    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.405499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.405472    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:34.419498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.419467    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"]
Oct 30 07:24:34.497319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.490075    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gng8h\" (UniqueName: \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.497319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.490265    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.497319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.490409    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.585768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.585731    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"]
Oct 30 07:24:34.589804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.589780    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.591438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.591412    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.591606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.591507    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.591673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.591617    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gng8h\" (UniqueName: \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.592205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.592181    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.594129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.594103    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.613144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.613114    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"]
Oct 30 07:24:34.617768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.617735    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gng8h\" (UniqueName: \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\") pod \"maas-api-revoke-sub-f60a44ed-pb6xj\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.693363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.693319    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmn5d\" (UniqueName: \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.693595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.693496    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.693717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.693697    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.716752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.716719    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:34.740801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.740769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:34.825521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.798168    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmn5d\" (UniqueName: \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.825521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.798382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.825521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.798597    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.825521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.815484    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.825521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.818821    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.833886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.833850    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmn5d\" (UniqueName: \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\") pod \"maas-api-revoke-sub-62ac8b76-w8mrf\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.917426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.917390    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:34.952769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.952733    6738 generic.go:356] "Generic (PLEG): container finished" podID="e2349be3-5aea-4165-8fa7-afd82b46caf4" containerID="2db7890e0e597871838a48946ea6c36334ac0cbcd2131ff87eb9ca92b46ac50e" exitCode=0
Oct 30 07:24:34.952992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:34.952804    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g" event={"ID":"e2349be3-5aea-4165-8fa7-afd82b46caf4","Type":"ContainerDied","Data":"2db7890e0e597871838a48946ea6c36334ac0cbcd2131ff87eb9ca92b46ac50e"}
Oct 30 07:24:35.031086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.031049    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"]
Oct 30 07:24:35.032414 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:35.032380    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d84550_81b0_4945_a6a3_df70b8cd4f85.slice/crio-7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e WatchSource:0}: Error finding container 7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e: Status 404 returned error can't find the container with id 7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e
Oct 30 07:24:35.259978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.259951    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"]
Oct 30 07:24:35.260415 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:35.260388    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445cec8f_7035_4bca_a3bd_4a064790caa2.slice/crio-88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f WatchSource:0}: Error finding container 88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f: Status 404 returned error can't find the container with id 88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f
Oct 30 07:24:35.961183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.961139    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf" event={"ID":"445cec8f-7035-4bca-a3bd-4a064790caa2","Type":"ContainerStarted","Data":"d5cf672c4841f2514bc7b90ab68e69f54c65bda62e2ad036440a19c0f6f9646c"}
Oct 30 07:24:35.961688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.961202    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf" event={"ID":"445cec8f-7035-4bca-a3bd-4a064790caa2","Type":"ContainerStarted","Data":"88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f"}
Oct 30 07:24:35.963021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.962995    6738 generic.go:356] "Generic (PLEG): container finished" podID="b8d84550-81b0-4945-a6a3-df70b8cd4f85" containerID="b4d6a32d8f51ab327c1902cd9cd5551a2c370f52a2206fc76f89f40debf740f1" exitCode=0
Oct 30 07:24:35.963147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.963055    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj" event={"ID":"b8d84550-81b0-4945-a6a3-df70b8cd4f85","Type":"ContainerDied","Data":"b4d6a32d8f51ab327c1902cd9cd5551a2c370f52a2206fc76f89f40debf740f1"}
Oct 30 07:24:35.963147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:35.963084    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj" event={"ID":"b8d84550-81b0-4945-a6a3-df70b8cd4f85","Type":"ContainerStarted","Data":"7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e"}
Oct 30 07:24:36.003731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.003670    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf" podStartSLOduration=2.003656069 podStartE2EDuration="2.003656069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:36.00178073 +0000 UTC m=+1636.434544624" watchObservedRunningTime="2026-10-30 07:24:36.003656069 +0000 UTC m=+1636.436419976"
Oct 30 07:24:36.130618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.130589    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:36.215825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.215797    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\") pod \"e2349be3-5aea-4165-8fa7-afd82b46caf4\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") "
Oct 30 07:24:36.216005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.215986    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\") pod \"e2349be3-5aea-4165-8fa7-afd82b46caf4\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") "
Oct 30 07:24:36.216099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.216030    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\" (UniqueName: \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\") pod \"e2349be3-5aea-4165-8fa7-afd82b46caf4\" (UID: \"e2349be3-5aea-4165-8fa7-afd82b46caf4\") "
Oct 30 07:24:36.216231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.216203    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca" pod "e2349be3-5aea-4165-8fa7-afd82b46caf4" (UID: "e2349be3-5aea-4165-8fa7-afd82b46caf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:36.216514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.216493    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:36.218107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.218081    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca" pod "e2349be3-5aea-4165-8fa7-afd82b46caf4" (UID: "e2349be3-5aea-4165-8fa7-afd82b46caf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:36.218192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.218150    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7" pod "e2349be3-5aea-4165-8fa7-afd82b46caf4" (UID: "e2349be3-5aea-4165-8fa7-afd82b46caf4"). InnerVolumeSpecName "kube-api-access-thmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:36.317667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.317632    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2349be3-5aea-4165-8fa7-afd82b46caf4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:36.317667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.317662    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmn7\" (UniqueName: \"kubernetes.io/projected/e2349be3-5aea-4165-8fa7-afd82b46caf4-kube-api-access-thmn7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:36.739398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.739064    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:36.975188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.975146    6738 generic.go:356] "Generic (PLEG): container finished" podID="445cec8f-7035-4bca-a3bd-4a064790caa2" containerID="d5cf672c4841f2514bc7b90ab68e69f54c65bda62e2ad036440a19c0f6f9646c" exitCode=0
Oct 30 07:24:36.975764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.975264    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf" event={"ID":"445cec8f-7035-4bca-a3bd-4a064790caa2","Type":"ContainerDied","Data":"d5cf672c4841f2514bc7b90ab68e69f54c65bda62e2ad036440a19c0f6f9646c"}
Oct 30 07:24:36.977137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.977107    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"
Oct 30 07:24:36.977283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.977135    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g" event={"ID":"e2349be3-5aea-4165-8fa7-afd82b46caf4","Type":"ContainerDied","Data":"a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b"}
Oct 30 07:24:36.977283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:36.977191    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9210391306ca889a8687337b71c21ff912d868ee780a6b6db0209cc7008fd3b"
Oct 30 07:24:37.205963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.205924    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:37.349402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.348592    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\") pod \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") "
Oct 30 07:24:37.349402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.348697    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\" (UniqueName: \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\") pod \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") "
Oct 30 07:24:37.349402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.349060    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\") pod \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\" (UID: \"b8d84550-81b0-4945-a6a3-df70b8cd4f85\") "
Oct 30 07:24:37.349974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.349946    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca" pod "b8d84550-81b0-4945-a6a3-df70b8cd4f85" (UID: "b8d84550-81b0-4945-a6a3-df70b8cd4f85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:37.351626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.351599    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h" pod "b8d84550-81b0-4945-a6a3-df70b8cd4f85" (UID: "b8d84550-81b0-4945-a6a3-df70b8cd4f85"). InnerVolumeSpecName "kube-api-access-gng8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:37.352241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.352208    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca" pod "b8d84550-81b0-4945-a6a3-df70b8cd4f85" (UID: "b8d84550-81b0-4945-a6a3-df70b8cd4f85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:37.451447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.451401    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:37.451447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.451444    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gng8h\" (UniqueName: \"kubernetes.io/projected/b8d84550-81b0-4945-a6a3-df70b8cd4f85-kube-api-access-gng8h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:37.451673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.451466    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d84550-81b0-4945-a6a3-df70b8cd4f85-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:37.628550 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:37.628509    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:24:37.996368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.994519    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"
Oct 30 07:24:37.996368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.995153    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj" event={"ID":"b8d84550-81b0-4945-a6a3-df70b8cd4f85","Type":"ContainerDied","Data":"7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e"}
Oct 30 07:24:37.996368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:37.995200    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a411e9d290f8e2308d2910324d9892ed0429c731a2133813931b025e9bae51e"
Oct 30 07:24:38.225875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.225848    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:38.362727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.362621    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\") pod \"445cec8f-7035-4bca-a3bd-4a064790caa2\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") "
Oct 30 07:24:38.362727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.362696    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\") pod \"445cec8f-7035-4bca-a3bd-4a064790caa2\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") "
Oct 30 07:24:38.362975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.362872    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\" (UniqueName: \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\") pod \"445cec8f-7035-4bca-a3bd-4a064790caa2\" (UID: \"445cec8f-7035-4bca-a3bd-4a064790caa2\") "
Oct 30 07:24:38.363031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.362976    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca" pod "445cec8f-7035-4bca-a3bd-4a064790caa2" (UID: "445cec8f-7035-4bca-a3bd-4a064790caa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:38.363331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.363275    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:38.364887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.364856    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca" pod "445cec8f-7035-4bca-a3bd-4a064790caa2" (UID: "445cec8f-7035-4bca-a3bd-4a064790caa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:38.364989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.364890    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d" pod "445cec8f-7035-4bca-a3bd-4a064790caa2" (UID: "445cec8f-7035-4bca-a3bd-4a064790caa2"). InnerVolumeSpecName "kube-api-access-zmn5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:38.464221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.464183    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmn5d\" (UniqueName: \"kubernetes.io/projected/445cec8f-7035-4bca-a3bd-4a064790caa2-kube-api-access-zmn5d\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:38.464221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.464214    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445cec8f-7035-4bca-a3bd-4a064790caa2-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:38.752140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:38.752094    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:39.002657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.002578    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"
Oct 30 07:24:39.002657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.002602    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf" event={"ID":"445cec8f-7035-4bca-a3bd-4a064790caa2","Type":"ContainerDied","Data":"88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f"}
Oct 30 07:24:39.002657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.002654    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c0d6bbd02f29cb5d465830764a8399be132d57bcfbe50aa3bf8e03c8a90a4f"
Oct 30 07:24:39.465728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.465698    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"]
Oct 30 07:24:39.466252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466234    6738 cpu_manager.go:414] "containerMap: removing container" podUID="445cec8f-7035-4bca-a3bd-4a064790caa2" containerName="revoke-keys"
Oct 30 07:24:39.466360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466255    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445cec8f-7035-4bca-a3bd-4a064790caa2" containerName="revoke-keys"
Oct 30 07:24:39.466360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466280    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e2349be3-5aea-4165-8fa7-afd82b46caf4" containerName="revoke-keys"
Oct 30 07:24:39.466360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466289    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2349be3-5aea-4165-8fa7-afd82b46caf4" containerName="revoke-keys"
Oct 30 07:24:39.466360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466341    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b8d84550-81b0-4945-a6a3-df70b8cd4f85" containerName="revoke-keys"
Oct 30 07:24:39.466360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466349    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d84550-81b0-4945-a6a3-df70b8cd4f85" containerName="revoke-keys"
Oct 30 07:24:39.466636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466474    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2349be3-5aea-4165-8fa7-afd82b46caf4" containerName="revoke-keys"
Oct 30 07:24:39.466636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466493    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d84550-81b0-4945-a6a3-df70b8cd4f85" containerName="revoke-keys"
Oct 30 07:24:39.466636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.466513    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="445cec8f-7035-4bca-a3bd-4a064790caa2" containerName="revoke-keys"
Oct 30 07:24:39.470038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.470015    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.488101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.488059    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"]
Oct 30 07:24:39.499344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.496505    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:39.499344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.496534    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:24:39.499344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.496534    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:24:39.499344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.496584    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:39.576847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.576808    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.577047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.576996    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq228\" (UniqueName: \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.577148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.577102    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.626148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.626123    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:24:39.677957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.677923    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq228\" (UniqueName: \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.678145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.677991    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.678145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.678044    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.678915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.678883    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.680633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.680607    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.702104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.702067    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq228\" (UniqueName: \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\") pod \"maas-api-revoke-sub-638c7b92-2k2vk\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.779417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779383    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779424    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779488    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779526    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779555    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779665    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779691    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779722    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779765    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\") pod \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\" (UID: \"4cc1d1cb-09d8-4d01-94e6-6af079b5c736\") "
Oct 30 07:24:39.779993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779791    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:39.779993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779827    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:39.779993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.779904    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:39.780214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780189    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:39.780261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780216    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:39.780376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780359    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.780434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780382    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.780434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780397    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.780434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780412    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.780434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.780429    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.782256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.782223    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:39.782256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.782230    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:39.782444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.782313    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "kube-api-access-wlfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:39.782444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.782336    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token" pod "4cc1d1cb-09d8-4d01-94e6-6af079b5c736" (UID: "4cc1d1cb-09d8-4d01-94e6-6af079b5c736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:39.787545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.787516    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:39.881612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.881530    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlfkq\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-kube-api-access-wlfkq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.881612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.881571    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.881612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.881587    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:39.881612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:39.881599    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc1d1cb-09d8-4d01-94e6-6af079b5c736-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:40.011656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.011619    6738 generic.go:356] "Generic (PLEG): container finished" podID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerID="9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151" exitCode=0
Oct 30 07:24:40.012089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.011664    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"
Oct 30 07:24:40.012089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.011705    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerDied","Data":"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"}
Oct 30 07:24:40.012089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.011762    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k" event={"ID":"4cc1d1cb-09d8-4d01-94e6-6af079b5c736","Type":"ContainerDied","Data":"b36ed3d5bdfb09e96f82f8e84297d1acce8c32f3e6559849c6cc9bb80da1de82"}
Oct 30 07:24:40.012089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.011791    6738 scope.go:122] "RemoveContainer" containerID="9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"
Oct 30 07:24:40.022704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.022679    6738 scope.go:122] "RemoveContainer" containerID="4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"
Oct 30 07:24:40.033696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.033673    6738 scope.go:122] "RemoveContainer" containerID="9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"
Oct 30 07:24:40.033964 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:40.033944    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151\": container with ID starting with 9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151 not found: ID does not exist" containerID="9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"
Oct 30 07:24:40.034031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.033973    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151"} err="failed to get container status \"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151\": rpc error: code = NotFound desc = could not find container \"9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151\": container with ID starting with 9aa3dee47e96f9bbe22d25aaac3b39f5d968022d0832b77f5f09715de1cd2151 not found: ID does not exist"
Oct 30 07:24:40.034031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.033990    6738 scope.go:122] "RemoveContainer" containerID="4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"
Oct 30 07:24:40.034209 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:40.034193    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25\": container with ID starting with 4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25 not found: ID does not exist" containerID="4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"
Oct 30 07:24:40.034248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.034215    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25"} err="failed to get container status \"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25\": rpc error: code = NotFound desc = could not find container \"4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25\": container with ID starting with 4808675ef48ee8f10893bbcb405dd4b4982f64105ac5a0c575f54e8061d46d25 not found: ID does not exist"
Oct 30 07:24:40.065009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.064972    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:24:40.077280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.077240    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-89664e-openshift-default-799c9cc465-4zh5k"]
Oct 30 07:24:40.101392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.101363    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"]
Oct 30 07:24:40.102020 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:40.101994    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc39cf6_3030_4e3d_a8e8_877dc5c88ac0.slice/crio-747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315 WatchSource:0}: Error finding container 747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315: Status 404 returned error can't find the container with id 747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315
Oct 30 07:24:40.535067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.534976    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:24:40.535239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.535192    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy" containerID="cri-o://3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888" gracePeriod=30
Oct 30 07:24:40.567411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:40.567378    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:41.023099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:41.023060    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk" event={"ID":"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0","Type":"ContainerStarted","Data":"fd2005044992417b74d5e5ef6b530bebb6f5bb3bde8b1c2deb6d4a02cc0b467d"}
Oct 30 07:24:41.023576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:41.023109    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk" event={"ID":"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0","Type":"ContainerStarted","Data":"747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315"}
Oct 30 07:24:41.066269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:41.066208    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk" podStartSLOduration=2.066190898 podStartE2EDuration="2.066190898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:41.063587834 +0000 UTC m=+1641.496351727" watchObservedRunningTime="2026-10-30 07:24:41.066190898 +0000 UTC m=+1641.498954790"
Oct 30 07:24:41.690273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:41.690239    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" path="/var/lib/kubelet/pods/4cc1d1cb-09d8-4d01-94e6-6af079b5c736/volumes"
Oct 30 07:24:42.031562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:42.031474    6738 generic.go:356] "Generic (PLEG): container finished" podID="9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" containerID="fd2005044992417b74d5e5ef6b530bebb6f5bb3bde8b1c2deb6d4a02cc0b467d" exitCode=0
Oct 30 07:24:42.031562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:42.031519    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk" event={"ID":"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0","Type":"ContainerDied","Data":"fd2005044992417b74d5e5ef6b530bebb6f5bb3bde8b1c2deb6d4a02cc0b467d"}
Oct 30 07:24:42.466036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:42.465994    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:43.191879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.191853    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:43.318029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.317934    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\" (UniqueName: \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\") pod \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") "
Oct 30 07:24:43.318029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.318013    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\") pod \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") "
Oct 30 07:24:43.318252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.318225    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\") pod \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\" (UID: \"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0\") "
Oct 30 07:24:43.318613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.318587    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca" pod "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" (UID: "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:43.320170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.320146    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca" pod "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" (UID: "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:43.320170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.320161    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228" pod "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" (UID: "9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0"). InnerVolumeSpecName "kube-api-access-rq228". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:43.394533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.394491    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"]
Oct 30 07:24:43.394999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.394982    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:43.394999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.394998    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:43.395152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.395046    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" containerName="revoke-keys"
Oct 30 07:24:43.395152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.395054    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" containerName="revoke-keys"
Oct 30 07:24:43.395152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.395138    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:43.395152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.395154    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:43.395401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.395165    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" containerName="revoke-keys"
Oct 30 07:24:43.398577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.398552    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.416631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.416599    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"]
Oct 30 07:24:43.419342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.419312    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:43.419342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.419339    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:43.419522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.419349    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq228\" (UniqueName: \"kubernetes.io/projected/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0-kube-api-access-rq228\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:43.520231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.520195    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.520422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.520257    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.520422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.520388    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.621328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.621232    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.621328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.621286    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.621526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.621345    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.621962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.621941    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.623662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.623630    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.644598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.644570    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\") pod \"maas-api-revoke-sub-07964a49-5jfv7\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:43.715470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:43.715432    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:44.019831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.019804    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"]
Oct 30 07:24:44.020805 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:44.020774    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceec2b0_1460_45f6_8f03_859720056a6e.slice/crio-d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff WatchSource:0}: Error finding container d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff: Status 404 returned error can't find the container with id d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff
Oct 30 07:24:44.047335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.047273    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7" event={"ID":"dceec2b0-1460-45f6-8f03-859720056a6e","Type":"ContainerStarted","Data":"d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff"}
Oct 30 07:24:44.048936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.048901    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk" event={"ID":"9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0","Type":"ContainerDied","Data":"747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315"}
Oct 30 07:24:44.048936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.048924    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"
Oct 30 07:24:44.049091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.048950    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="747a4dec31d3072714390d840577af633c8f7d7c63d7ac6a22acf60ba0bae315"
Oct 30 07:24:44.337622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:44.337582    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:45.057275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.057183    6738 generic.go:356] "Generic (PLEG): container finished" podID="dceec2b0-1460-45f6-8f03-859720056a6e" containerID="eb8215e441667769622c792bb8098521f7b1dba45467c8f0471376b9aa143e2e" exitCode=0
Oct 30 07:24:45.057275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.057263    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7" event={"ID":"dceec2b0-1460-45f6-8f03-859720056a6e","Type":"ContainerDied","Data":"eb8215e441667769622c792bb8098521f7b1dba45467c8f0471376b9aa143e2e"}
Oct 30 07:24:45.743037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.743002    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:24:45.743810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.743776    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:45.743810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.743805    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc1d1cb-09d8-4d01-94e6-6af079b5c736" containerName="istio-proxy"
Oct 30 07:24:45.751944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.751914    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.771488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.771456    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:24:45.814094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.814025    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:24:45.854957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.854915    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.854966    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855113    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855191    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrl6w\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855262    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855306    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855421    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855445    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.855531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.855468    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.956395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956352    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956406    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956447    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956488    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956574    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956625    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956653    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956654    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:45.956868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956717    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.956868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.956860    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\") pod \"266dea85-fd34-4eba-a344-f07f6fed8b8b\" (UID: \"266dea85-fd34-4eba-a344-f07f6fed8b8b\") "
Oct 30 07:24:45.957147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957023    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957030    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:45.957147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957120    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957228    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:45.957520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957353    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957419    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957529    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrl6w\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957534    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957657    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957683    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:45.957717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957709    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957781    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957805    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.957973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957880    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:45.957973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957893    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:45.958190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.957985    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:45.958190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.958138    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.958190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.958157    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.958625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.958601    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.958781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.958766    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:45.958872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.958861    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266dea85-fd34-4eba-a344-f07f6fed8b8b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:45.959372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.959328    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.959477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.959385    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:45.959611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.959526    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:45.959611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.959587    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:45.960233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.960210    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.960439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.960410    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs" pod "266dea85-fd34-4eba-a344-f07f6fed8b8b" (UID: "266dea85-fd34-4eba-a344-f07f6fed8b8b"). InnerVolumeSpecName "kube-api-access-hb7vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:45.961492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.961470    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.981429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.981392    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrl6w\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:45.981429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:45.981413    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:46.060091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.060057    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.060091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.060086    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.060091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.060096    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.060365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.060103    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266dea85-fd34-4eba-a344-f07f6fed8b8b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.060365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.060112    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb7vs\" (UniqueName: \"kubernetes.io/projected/266dea85-fd34-4eba-a344-f07f6fed8b8b-kube-api-access-hb7vs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.066590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.066502    6738 generic.go:356] "Generic (PLEG): container finished" podID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerID="3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888" exitCode=0
Oct 30 07:24:46.066590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.066566    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"
Oct 30 07:24:46.066590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.066579    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" event={"ID":"266dea85-fd34-4eba-a344-f07f6fed8b8b","Type":"ContainerDied","Data":"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"}
Oct 30 07:24:46.066869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.066630    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh" event={"ID":"266dea85-fd34-4eba-a344-f07f6fed8b8b","Type":"ContainerDied","Data":"f82fd85a48b7fffce01bc74f74010f9e5adde19d3d589ef90e7de1ed4d2e120e"}
Oct 30 07:24:46.066869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.066651    6738 scope.go:122] "RemoveContainer" containerID="3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"
Oct 30 07:24:46.074334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.074272    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:46.079904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.079878    6738 scope.go:122] "RemoveContainer" containerID="3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"
Oct 30 07:24:46.080264 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:24:46.080243    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888\": container with ID starting with 3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888 not found: ID does not exist" containerID="3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"
Oct 30 07:24:46.080366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.080274    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888"} err="failed to get container status \"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888\": rpc error: code = NotFound desc = could not find container \"3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888\": container with ID starting with 3c5498ff62f340160ffa3cf3be11efd5a9ab9930193246577ba1f60683d8f888 not found: ID does not exist"
Oct 30 07:24:46.154033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.151891    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:24:46.159599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.159324    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pt8fh"]
Oct 30 07:24:46.242125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.242090    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:46.269162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.269126    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\") pod \"dceec2b0-1460-45f6-8f03-859720056a6e\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") "
Oct 30 07:24:46.269405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.269209    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\") pod \"dceec2b0-1460-45f6-8f03-859720056a6e\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") "
Oct 30 07:24:46.269405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.269240    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\") pod \"dceec2b0-1460-45f6-8f03-859720056a6e\" (UID: \"dceec2b0-1460-45f6-8f03-859720056a6e\") "
Oct 30 07:24:46.270046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.270018    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca" pod "dceec2b0-1460-45f6-8f03-859720056a6e" (UID: "dceec2b0-1460-45f6-8f03-859720056a6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:46.272013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.271986    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx" pod "dceec2b0-1460-45f6-8f03-859720056a6e" (UID: "dceec2b0-1460-45f6-8f03-859720056a6e"). InnerVolumeSpecName "kube-api-access-s64bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:46.272013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.271992    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca" pod "dceec2b0-1460-45f6-8f03-859720056a6e" (UID: "dceec2b0-1460-45f6-8f03-859720056a6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:46.312111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.312067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:46.386350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.385969    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.386350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.386014    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/dceec2b0-1460-45f6-8f03-859720056a6e-kube-api-access-s64bx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.386350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.386030    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dceec2b0-1460-45f6-8f03-859720056a6e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:46.450338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.450274    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:24:46.450872 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:46.450845    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b14404c_2018_4874_a3da_e61218f4166a.slice/crio-0c56718c6704f5da4ed08e499da1654f473a53f4a82d5157b2c709c5666a4137 WatchSource:0}: Error finding container 0c56718c6704f5da4ed08e499da1654f473a53f4a82d5157b2c709c5666a4137: Status 404 returned error can't find the container with id 0c56718c6704f5da4ed08e499da1654f473a53f4a82d5157b2c709c5666a4137
Oct 30 07:24:46.453280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.453243    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:46.453401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.453329    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:46.453401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:46.453390    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:47.099942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.099912    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"
Oct 30 07:24:47.100473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.099947    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7" event={"ID":"dceec2b0-1460-45f6-8f03-859720056a6e","Type":"ContainerDied","Data":"d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff"}
Oct 30 07:24:47.100473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.100005    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d773256c643715196e6517fc77808eef561aa1d3ca22fb474037381a9ee0b5ff"
Oct 30 07:24:47.101791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.101757    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" event={"ID":"4b14404c-2018-4874-a3da-e61218f4166a","Type":"ContainerStarted","Data":"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"}
Oct 30 07:24:47.101904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.101808    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" event={"ID":"4b14404c-2018-4874-a3da-e61218f4166a","Type":"ContainerStarted","Data":"0c56718c6704f5da4ed08e499da1654f473a53f4a82d5157b2c709c5666a4137"}
Oct 30 07:24:47.148475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.148409    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podStartSLOduration=2.148392546 podStartE2EDuration="2.148392546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:47.146711642 +0000 UTC m=+1647.579475535" watchObservedRunningTime="2026-10-30 07:24:47.148392546 +0000 UTC m=+1647.581156496"
Oct 30 07:24:47.696755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:47.696722    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" path="/var/lib/kubelet/pods/266dea85-fd34-4eba-a344-f07f6fed8b8b/volumes"
Oct 30 07:24:48.088378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.078675    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:48.092466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.092422    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:48.092660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.092503    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:48.107971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.107935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:48.993316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.992563    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"]
Oct 30 07:24:48.993541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993339    6738 cpu_manager.go:414] "containerMap: removing container" podUID="dceec2b0-1460-45f6-8f03-859720056a6e" containerName="revoke-keys"
Oct 30 07:24:48.993541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993358    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dceec2b0-1460-45f6-8f03-859720056a6e" containerName="revoke-keys"
Oct 30 07:24:48.993541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993385    6738 cpu_manager.go:414] "containerMap: removing container" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy"
Oct 30 07:24:48.993541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993393    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy"
Oct 30 07:24:48.993796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993556    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="dceec2b0-1460-45f6-8f03-859720056a6e" containerName="revoke-keys"
Oct 30 07:24:48.993796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.993575    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="266dea85-fd34-4eba-a344-f07f6fed8b8b" containerName="istio-proxy"
Oct 30 07:24:48.998548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:48.998511    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.005106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.005075    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"]
Oct 30 07:24:49.017581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.017541    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.017895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.017869    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.018063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.018044    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgmg\" (UniqueName: \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.031020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.030993    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:24:49.031161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.031044    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:49.031205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.031157    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:49.031205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.031162    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:24:49.087836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.087794    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:49.088148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.088108    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:49.119082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.119042    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.119566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.119130    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.119566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.119177    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgmg\" (UniqueName: \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.119863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.119842    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.121499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.121481    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.143074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.143037    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgmg\" (UniqueName: \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\") pod \"maas-api-revoke-sub-b3071736-2kzsm\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.328386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.328272    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:49.700023 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:49.699978    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8e635e_9d58_43da_9719_554636c934eb.slice/crio-2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b WatchSource:0}: Error finding container 2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b: Status 404 returned error can't find the container with id 2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b
Oct 30 07:24:49.704201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:49.704177    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"]
Oct 30 07:24:50.085393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.076796    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:50.085393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.076867    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:50.123861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.123824    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm" event={"ID":"2a8e635e-9d58-43da-9719-554636c934eb","Type":"ContainerStarted","Data":"36c17d1dc8b58ba97f1309f7fa5a96e32f3828d03a74dfb24d729dea62e7ab66"}
Oct 30 07:24:50.124276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.123922    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm" event={"ID":"2a8e635e-9d58-43da-9719-554636c934eb","Type":"ContainerStarted","Data":"2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b"}
Oct 30 07:24:50.191318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.190437    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm" podStartSLOduration=2.190416146 podStartE2EDuration="2.190416146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:50.165725654 +0000 UTC m=+1650.598489546" watchObservedRunningTime="2026-10-30 07:24:50.190416146 +0000 UTC m=+1650.623180040"
Oct 30 07:24:50.191515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:50.191337    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:51.075385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:51.074747    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:51.075385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:51.074810    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:51.132503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:51.132463    6738 generic.go:356] "Generic (PLEG): container finished" podID="2a8e635e-9d58-43da-9719-554636c934eb" containerID="36c17d1dc8b58ba97f1309f7fa5a96e32f3828d03a74dfb24d729dea62e7ab66" exitCode=0
Oct 30 07:24:51.132909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:51.132556    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm" event={"ID":"2a8e635e-9d58-43da-9719-554636c934eb","Type":"ContainerDied","Data":"36c17d1dc8b58ba97f1309f7fa5a96e32f3828d03a74dfb24d729dea62e7ab66"}
Oct 30 07:24:52.092356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.092017    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:52.092356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.092171    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:52.092356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.092224    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:52.349410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.349345    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:52.434100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.434062    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:24:52.434810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.434790    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2a8e635e-9d58-43da-9719-554636c934eb" containerName="revoke-keys"
Oct 30 07:24:52.434882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.434813    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8e635e-9d58-43da-9719-554636c934eb" containerName="revoke-keys"
Oct 30 07:24:52.434931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.434921    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8e635e-9d58-43da-9719-554636c934eb" containerName="revoke-keys"
Oct 30 07:24:52.438375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.438353    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.454650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454623    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\" (UniqueName: \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\") pod \"2a8e635e-9d58-43da-9719-554636c934eb\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") "
Oct 30 07:24:52.454809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454683    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\") pod \"2a8e635e-9d58-43da-9719-554636c934eb\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") "
Oct 30 07:24:52.454809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454768    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\") pod \"2a8e635e-9d58-43da-9719-554636c934eb\" (UID: \"2a8e635e-9d58-43da-9719-554636c934eb\") "
Oct 30 07:24:52.454926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454880    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.454926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454901    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.454947    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca" pod "2a8e635e-9d58-43da-9719-554636c934eb" (UID: "2a8e635e-9d58-43da-9719-554636c934eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:52.455134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455115    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455159    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455215    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vrx\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455269    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455386    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455453    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455494    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.455677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.455662    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:52.457064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.457036    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg" pod "2a8e635e-9d58-43da-9719-554636c934eb" (UID: "2a8e635e-9d58-43da-9719-554636c934eb"). InnerVolumeSpecName "kube-api-access-mvgmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:52.457567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.457540    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca" pod "2a8e635e-9d58-43da-9719-554636c934eb" (UID: "2a8e635e-9d58-43da-9719-554636c934eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:52.462484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.462453    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:24:52.466522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.466401    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:24:52.471706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.471676    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.503086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.503054    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:24:52.557051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557005    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557072    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557154    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557210    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557262    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557323    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557375    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557471    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557514    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557614    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vrx\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557645    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557684    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrkxw\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.557797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557756    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557799    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557848    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.557905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557885    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.558078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.557926    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.558078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558015    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8e635e-9d58-43da-9719-554636c934eb-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:52.558078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558034    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgmg\" (UniqueName: \"kubernetes.io/projected/2a8e635e-9d58-43da-9719-554636c934eb-kube-api-access-mvgmg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:24:52.558235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558077    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.558235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558125    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.558235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558185    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.558444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.558323    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.559979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.559955    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.560181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.560158    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.581894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.581859    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vrx\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.582231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.582208    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.659227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659173    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659255    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659328    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrkxw\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659384    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659498    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659537    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659570    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659600    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659659    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.659949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.659932    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.660131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.660101    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.660268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.660222    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.660394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.660315    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.660639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.660617    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.661923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.661903    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.662485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.662468    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.682891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.682859    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.683100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.683074    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrkxw\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:52.752390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.752354    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:52.813604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:52.813452    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:53.075402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.075365    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 30 07:24:53.075610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.075429    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 30 07:24:53.077639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.077610    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:24:53.079653 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:53.079625    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7b524_2ec7_46f2_ae32_c33291319f81.slice/crio-ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2 WatchSource:0}: Error finding container ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2: Status 404 returned error can't find the container with id ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2
Oct 30 07:24:53.082660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.082623    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.082770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.082708    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.082844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.082778    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.165844 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:53.165801    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de7d078_4ca1_4141_a97e_030055b71ca1.slice/crio-7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360 WatchSource:0}: Error finding container 7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360: Status 404 returned error can't find the container with id 7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360
Oct 30 07:24:53.166344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.166318    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:24:53.166616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.166585    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" event={"ID":"a5c7b524-2ec7-46f2-ae32-c33291319f81","Type":"ContainerStarted","Data":"ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2"}
Oct 30 07:24:53.168240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168202    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.168389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168281    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.168389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168344    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:24:53.168525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168399    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm" event={"ID":"2a8e635e-9d58-43da-9719-554636c934eb","Type":"ContainerDied","Data":"2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b"}
Oct 30 07:24:53.168525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168450    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b1a387e8a122c7fb2a700e412dbb1bad46ab08e51eaff219fcef08c48d2aa8b"
Oct 30 07:24:53.168525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.168516    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"
Oct 30 07:24:53.816171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.816134    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"]
Oct 30 07:24:53.820873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.820840    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:53.839015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.838971    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"]
Oct 30 07:24:53.845864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.845464    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:53.845864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.845522    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:53.845864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.845571    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:24:53.845864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.845682    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:24:53.992863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.992694    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:53.993077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.992996    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:53.995745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:53.994498    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qw8\" (UniqueName: \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.060238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.060202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:54.092384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.091626    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:54.096314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.093466    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:54.096507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.096462    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.096583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.096543    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qw8\" (UniqueName: \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.099322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.096746    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.099322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.097593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.123242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.106615    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:24:54.123242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.108476    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.156718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.142210    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qw8\" (UniqueName: \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\") pod \"maas-api-revoke-sub-276c87e9-hbjzg\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.254635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.254585    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" event={"ID":"a5c7b524-2ec7-46f2-ae32-c33291319f81","Type":"ContainerStarted","Data":"0cad1a6fe6f701991a29ea38edf7d8486a199bbed4ae01c521d5cfdd8025ea03"}
Oct 30 07:24:54.338411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.331322    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" event={"ID":"6de7d078-4ca1-4141-a97e-030055b71ca1","Type":"ContainerStarted","Data":"f5aea6022f24fa0a884778ef2fcc5efdce975024f4de9182a55c09464aa1ee15"}
Oct 30 07:24:54.338411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.332602    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" event={"ID":"6de7d078-4ca1-4141-a97e-030055b71ca1","Type":"ContainerStarted","Data":"7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360"}
Oct 30 07:24:54.339520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.339456    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podStartSLOduration=2.339435824 podStartE2EDuration="2.339435824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:54.321783885 +0000 UTC m=+1654.754547778" watchObservedRunningTime="2026-10-30 07:24:54.339435824 +0000 UTC m=+1654.772199730"
Oct 30 07:24:54.424211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.424144    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podStartSLOduration=2.424127767 podStartE2EDuration="2.424127767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:54.419640459 +0000 UTC m=+1654.852404354" watchObservedRunningTime="2026-10-30 07:24:54.424127767 +0000 UTC m=+1654.856891730"
Oct 30 07:24:54.440353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.440323    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:24:54.752854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.752759    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:24:54.754333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.754305    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:54.754493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.754364    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:54.815437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.812035    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"]
Oct 30 07:24:54.815437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.815029    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:24:54.825491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.820455    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:54.825491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.820527    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:24:54.835129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.835105    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"]
Oct 30 07:24:54.839995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.839974    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:54.856538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.856510    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"]
Oct 30 07:24:54.939355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.939317    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:54.939514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.939397    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:54.939514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:54.939432    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqsw\" (UniqueName: \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.040339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.040211    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.040339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.040282    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.040560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.040356    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqsw\" (UniqueName: \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.041064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.041041    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.091333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.089969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqsw\" (UniqueName: \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.091333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.090287    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525d33eb-4vh4v\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.231391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.231354    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:24:55.347658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.347530    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg" event={"ID":"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54","Type":"ContainerStarted","Data":"b4e01827f73d3fe0334a0c82f6a9aeedd357c30f4e5b433a7ad19d3145778ebe"}
Oct 30 07:24:55.661139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.660825    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"]
Oct 30 07:24:55.703342 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:55.697877    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370bba0e_f2cf_42aa_a5c4_e62ad213d627.slice/crio-dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab WatchSource:0}: Error finding container dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab: Status 404 returned error can't find the container with id dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab
Oct 30 07:24:55.886626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.880959    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:55.886626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.881045    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:24:55.898924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.898875    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:55.899272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:55.899173    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:56.398390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.398351    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v" event={"ID":"370bba0e-f2cf-42aa-a5c4-e62ad213d627","Type":"ContainerStarted","Data":"dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab"}
Oct 30 07:24:56.410213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.410179    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg" event={"ID":"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54","Type":"ContainerStarted","Data":"42acfe19597c9ee782db08a39cd771415e9b931c9eefa2087eb6f0414296b1db"}
Oct 30 07:24:56.467065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.466968    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg" podStartSLOduration=3.46693457 podStartE2EDuration="3.46693457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:56.463860488 +0000 UTC m=+1656.896624382" watchObservedRunningTime="2026-10-30 07:24:56.46693457 +0000 UTC m=+1656.899698464"
Oct 30 07:24:56.763219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.763159    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:56.763566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.763526    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:56.816907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.816869    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:56.817241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.817194    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:24:56.908560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.908451    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:56.951307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.951238    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"]
Oct 30 07:24:56.968330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.965721    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:56.986282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:56.986254    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"]
Oct 30 07:24:57.068317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.065354    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrhp\" (UniqueName: \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.093320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.069409    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.093320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.082511    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.121321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.108529    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.202384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.186141    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.202384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.186269    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrhp\" (UniqueName: \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.202384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.186328    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.202384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.202256    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.262117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.262060    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.288937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.278960    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.421470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.419326    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrhp\" (UniqueName: \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\") pod \"maas-api-revoke-sub-4a029f64-62789\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.504320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.481460    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.522889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.521932    6738 generic.go:356] "Generic (PLEG): container finished" podID="be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" containerID="42acfe19597c9ee782db08a39cd771415e9b931c9eefa2087eb6f0414296b1db" exitCode=0
Oct 30 07:24:57.522889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.522083    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg" event={"ID":"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54","Type":"ContainerDied","Data":"42acfe19597c9ee782db08a39cd771415e9b931c9eefa2087eb6f0414296b1db"}
Oct 30 07:24:57.543321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.528624    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v" event={"ID":"370bba0e-f2cf-42aa-a5c4-e62ad213d627","Type":"ContainerStarted","Data":"4b8df58444ec265a268975198300a2b39ba990eb5f30b722012048c2c953628c"}
Oct 30 07:24:57.563474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.563422    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.655011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.654943    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v" podStartSLOduration=3.654920131 podStartE2EDuration="3.654920131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:57.639962128 +0000 UTC m=+1658.072726076" watchObservedRunningTime="2026-10-30 07:24:57.654920131 +0000 UTC m=+1658.087684025"
Oct 30 07:24:57.701831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.701508    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:24:57.794423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.777457    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.798315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.794802    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:57.798315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.794870    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:57.851268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.851226    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:24:57.876438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.876403    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:57.876727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.876697    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:24:57.919482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.919437    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.920545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:57.920520    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:24:58.142378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.124575    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.142378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.125100    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:24:58.183657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.183619    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:24:58.267117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.266857    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.269894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.216138    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 07:24:58.269894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.216102    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:24:58.269894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.268835    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:24:58.523004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.502395    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:24:58.523004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.502931    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:24:58.596776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.592001    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.596776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.592633    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:24:58.839321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.771047    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:58.839321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.771104    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:58.839321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.823264    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:58.893320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.887757    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:24:58.936320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.915656    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.936320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.916080    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:24:58.936320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.933231    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:59.008320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:58.997649    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:24:59.088320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.077049    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:24:59.182321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.182026    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:59.297215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.297168    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:59.351197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.309532    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:24:59.352012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.351988    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:24:59.362398 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:24:59.362347    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2c2274_b173_457e_985c_6d5b9f871a7e.slice/crio-5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c WatchSource:0}: Error finding container 5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c: Status 404 returned error can't find the container with id 5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c
Oct 30 07:24:59.406319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.386699    6738 generic.go:356] "Generic (PLEG): container finished" podID="370bba0e-f2cf-42aa-a5c4-e62ad213d627" containerID="4b8df58444ec265a268975198300a2b39ba990eb5f30b722012048c2c953628c" exitCode=0
Oct 30 07:24:59.406319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.387665    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v" event={"ID":"370bba0e-f2cf-42aa-a5c4-e62ad213d627","Type":"ContainerDied","Data":"4b8df58444ec265a268975198300a2b39ba990eb5f30b722012048c2c953628c"}
Oct 30 07:24:59.464529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.464422    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"]
Oct 30 07:24:59.521065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.521009    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:59.563782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.563749    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:24:59.664186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.664158    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:24:59.682019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.681983    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:24:59.756680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.756210    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:24:59.756680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.756270    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:24:59.761608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.761585    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:24:59.762462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.762046    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:24:59.832536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.832500    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:24:59.832908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:24:59.832878    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:25:00.041595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.041565    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:25:00.066894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.066857    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\" (UniqueName: \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\") pod \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") "
Oct 30 07:25:00.067075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.066914    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\") pod \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") "
Oct 30 07:25:00.076328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.067285    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca" pod "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" (UID: "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:00.127168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.127103    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8" pod "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" (UID: "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54"). InnerVolumeSpecName "kube-api-access-s4qw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:00.184223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.183088    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\") pod \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\" (UID: \"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54\") "
Oct 30 07:25:00.184223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.183781    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qw8\" (UniqueName: \"kubernetes.io/projected/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-kube-api-access-s4qw8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:00.184223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.183810    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:00.213379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.206038    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca" pod "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" (UID: "be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:00.311377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.310779    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:00.418836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.418791    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"
Oct 30 07:25:00.419174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.419140    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg" event={"ID":"be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54","Type":"ContainerDied","Data":"b4e01827f73d3fe0334a0c82f6a9aeedd357c30f4e5b433a7ad19d3145778ebe"}
Oct 30 07:25:00.419403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.419385    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e01827f73d3fe0334a0c82f6a9aeedd357c30f4e5b433a7ad19d3145778ebe"
Oct 30 07:25:00.422511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.422483    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789" event={"ID":"3b2c2274-b173-457e-985c-6d5b9f871a7e","Type":"ContainerStarted","Data":"4b03c8d9ccadff45fdecd74924c6a555ab4e55f22f20d14aa1bb83a6d7d6c3f9"}
Oct 30 07:25:00.422635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.422533    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789" event={"ID":"3b2c2274-b173-457e-985c-6d5b9f871a7e","Type":"ContainerStarted","Data":"5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c"}
Oct 30 07:25:00.466358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.464323    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789" podStartSLOduration=4.464288425 podStartE2EDuration="4.464288425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:00.462491297 +0000 UTC m=+1660.895255191" watchObservedRunningTime="2026-10-30 07:25:00.464288425 +0000 UTC m=+1660.897052318"
Oct 30 07:25:00.727341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.727279    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:25:00.753495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.753457    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:25:00.753717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.753518    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:25:00.827740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.827027    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:25:00.827740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.827086    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:25:00.829463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.828452    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\" (UniqueName: \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\") pod \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") "
Oct 30 07:25:00.829463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.828673    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\") pod \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") "
Oct 30 07:25:00.829463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.828727    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\") pod \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\" (UID: \"370bba0e-f2cf-42aa-a5c4-e62ad213d627\") "
Oct 30 07:25:00.830149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.829847    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca" pod "370bba0e-f2cf-42aa-a5c4-e62ad213d627" (UID: "370bba0e-f2cf-42aa-a5c4-e62ad213d627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:00.840542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.839641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca" pod "370bba0e-f2cf-42aa-a5c4-e62ad213d627" (UID: "370bba0e-f2cf-42aa-a5c4-e62ad213d627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:00.842991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.842935    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:25:00.844009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.843984    6738 cpu_manager.go:414] "containerMap: removing container" podUID="be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" containerName="revoke-keys"
Oct 30 07:25:00.844171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.844159    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" containerName="revoke-keys"
Oct 30 07:25:00.844289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.844279    6738 cpu_manager.go:414] "containerMap: removing container" podUID="370bba0e-f2cf-42aa-a5c4-e62ad213d627" containerName="revoke-keys"
Oct 30 07:25:00.844387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.844376    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370bba0e-f2cf-42aa-a5c4-e62ad213d627" containerName="revoke-keys"
Oct 30 07:25:00.844616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.844604    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="370bba0e-f2cf-42aa-a5c4-e62ad213d627" containerName="revoke-keys"
Oct 30 07:25:00.844712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.844703    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" containerName="revoke-keys"
Oct 30 07:25:00.848714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.848664    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw" pod "370bba0e-f2cf-42aa-a5c4-e62ad213d627" (UID: "370bba0e-f2cf-42aa-a5c4-e62ad213d627"). InnerVolumeSpecName "kube-api-access-jsqsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:00.852751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.852719    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:25:00.852946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.852846    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.929725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.929645    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.929918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.929900    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930033    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930131    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930383    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930580    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930626    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930713    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcdz\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:00.930852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930837    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:00.930921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930855    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370bba0e-f2cf-42aa-a5c4-e62ad213d627-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:00.930921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:00.930865    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsqsw\" (UniqueName: \"kubernetes.io/projected/370bba0e-f2cf-42aa-a5c4-e62ad213d627-kube-api-access-jsqsw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:01.031607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031561    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.031909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031624    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.031909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031659    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.031909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031823    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.031909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031855    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.031909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031894    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcdz\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.031994    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.032016    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.032040    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.032417    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.032838    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.032982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.032864    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.033166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.033145    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.033777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.033752    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.036113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.036089    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.036442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.036423    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.054997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.054958    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.055557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.055532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcdz\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.098254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.098211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:01.302196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.302095    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:01.436863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.436826    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"
Oct 30 07:25:01.437084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.436826    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v" event={"ID":"370bba0e-f2cf-42aa-a5c4-e62ad213d627","Type":"ContainerDied","Data":"dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab"}
Oct 30 07:25:01.437084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.436988    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc3e2e74ef3f37b7811ea99105950bca93589bdd9b53bd8bf1577809804736ab"
Oct 30 07:25:01.439008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.438978    6738 generic.go:356] "Generic (PLEG): container finished" podID="3b2c2274-b173-457e-985c-6d5b9f871a7e" containerID="4b03c8d9ccadff45fdecd74924c6a555ab4e55f22f20d14aa1bb83a6d7d6c3f9" exitCode=0
Oct 30 07:25:01.439155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.439064    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789" event={"ID":"3b2c2274-b173-457e-985c-6d5b9f871a7e","Type":"ContainerDied","Data":"4b03c8d9ccadff45fdecd74924c6a555ab4e55f22f20d14aa1bb83a6d7d6c3f9"}
Oct 30 07:25:01.657420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.657377    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:25:01.657852 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:01.657827    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612fdfbc_d52e_45b3_9d55_67448f490e4a.slice/crio-2e7274a372932d9708a3e8d9e38a8d0b7b5d137061fa4845a48032296c80e61b WatchSource:0}: Error finding container 2e7274a372932d9708a3e8d9e38a8d0b7b5d137061fa4845a48032296c80e61b: Status 404 returned error can't find the container with id 2e7274a372932d9708a3e8d9e38a8d0b7b5d137061fa4845a48032296c80e61b
Oct 30 07:25:01.661397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.661353    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:01.661483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.661447    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:01.661532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.661497    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:01.752905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.752865    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 07:25:01.753080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.752932    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 07:25:01.814314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.814253    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 07:25:01.814529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:01.814338    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 07:25:02.460824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.460776    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" event={"ID":"612fdfbc-d52e-45b3-9d55-67448f490e4a","Type":"ContainerStarted","Data":"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"}
Oct 30 07:25:02.461343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.460884    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" event={"ID":"612fdfbc-d52e-45b3-9d55-67448f490e4a","Type":"ContainerStarted","Data":"2e7274a372932d9708a3e8d9e38a8d0b7b5d137061fa4845a48032296c80e61b"}
Oct 30 07:25:02.528314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.528138    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podStartSLOduration=2.528120936 podStartE2EDuration="2.528120936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:02.525007623 +0000 UTC m=+1662.957771517" watchObservedRunningTime="2026-10-30 07:25:02.528120936 +0000 UTC m=+1662.960884829"
Oct 30 07:25:02.714360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.714254    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:25:02.757027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.756988    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\") pod \"3b2c2274-b173-457e-985c-6d5b9f871a7e\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") "
Oct 30 07:25:02.757247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.757229    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\" (UniqueName: \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\") pod \"3b2c2274-b173-457e-985c-6d5b9f871a7e\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") "
Oct 30 07:25:02.757357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.757278    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\") pod \"3b2c2274-b173-457e-985c-6d5b9f871a7e\" (UID: \"3b2c2274-b173-457e-985c-6d5b9f871a7e\") "
Oct 30 07:25:02.757466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.757436    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca" pod "3b2c2274-b173-457e-985c-6d5b9f871a7e" (UID: "3b2c2274-b173-457e-985c-6d5b9f871a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:02.757822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.757794    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:02.764684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.760896    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp" pod "3b2c2274-b173-457e-985c-6d5b9f871a7e" (UID: "3b2c2274-b173-457e-985c-6d5b9f871a7e"). InnerVolumeSpecName "kube-api-access-qvrhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:02.764684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.761234    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca" pod "3b2c2274-b173-457e-985c-6d5b9f871a7e" (UID: "3b2c2274-b173-457e-985c-6d5b9f871a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:02.858835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.858796    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2c2274-b173-457e-985c-6d5b9f871a7e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:02.859006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:02.858840    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrhp\" (UniqueName: \"kubernetes.io/projected/3b2c2274-b173-457e-985c-6d5b9f871a7e-kube-api-access-qvrhp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:03.249355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.249030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:03.309318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.307703    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:03.331316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.319452    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:03.331316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.319516    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:03.503370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.502700    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"
Oct 30 07:25:03.517154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.516627    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789" event={"ID":"3b2c2274-b173-457e-985c-6d5b9f871a7e","Type":"ContainerDied","Data":"5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c"}
Oct 30 07:25:03.517154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.516708    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be8688e1d5e6c0fef57150bbbdaeddf28a9bbcfb15f1acd2af58a4a9aa23f6c"
Oct 30 07:25:03.801375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.800435    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:03.801375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.800508    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:03.857464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.857413    6738 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:03.857656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:03.857508    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:04.161100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.161056    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:25:04.162542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.162518    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:25:04.189318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.187665    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:25:04.189318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.188938    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:25:04.210325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.209536    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:25:04.244947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.244908    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:25:04.303110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.303071    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:04.303333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:04.303139    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:05.239761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.239724    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"]
Oct 30 07:25:05.240459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.240435    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3b2c2274-b173-457e-985c-6d5b9f871a7e" containerName="revoke-keys"
Oct 30 07:25:05.240459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.240457    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2c2274-b173-457e-985c-6d5b9f871a7e" containerName="revoke-keys"
Oct 30 07:25:05.240647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.240552    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2c2274-b173-457e-985c-6d5b9f871a7e" containerName="revoke-keys"
Oct 30 07:25:05.243566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.243545    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.264407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.264376    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"]
Oct 30 07:25:05.284324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.282764    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:25:05.284324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.282843    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:25:05.284324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.282906    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:25:05.284324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.283004    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:25:05.302853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.302810    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:05.303068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.302862    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:05.339935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.339902    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wln5t\" (UniqueName: \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.340114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.340041    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.340114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.340084    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.431667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.431631    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:05.441009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.440978    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.441154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.441015    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.441224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.441151    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wln5t\" (UniqueName: \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.441927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.441904    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.443814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.443789    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.487319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.487193    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wln5t\" (UniqueName: \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\") pod \"maas-api-revoke-sub-deffd453-fxjk6\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.582402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.581703    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:05.906303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:05.906257    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"]
Oct 30 07:25:05.908052 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:05.908023    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33380a5_dac3_4efb_90de_7de883744c04.slice/crio-c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926 WatchSource:0}: Error finding container c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926: Status 404 returned error can't find the container with id c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926
Oct 30 07:25:06.303240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.303210    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:06.303665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.303277    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:06.509531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.509497    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:06.551775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.551680    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6" event={"ID":"f33380a5-dac3-4efb-90de-7de883744c04","Type":"ContainerStarted","Data":"68db72e00ae6b8215b3294e8db9eaff675addcd62d1c4f881d9c21c6f0a68ae7"}
Oct 30 07:25:06.551775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.551736    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6" event={"ID":"f33380a5-dac3-4efb-90de-7de883744c04","Type":"ContainerStarted","Data":"c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926"}
Oct 30 07:25:06.596503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.596434    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6" podStartSLOduration=1.596418481 podStartE2EDuration="1.596418481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:06.592728674 +0000 UTC m=+1667.025492567" watchObservedRunningTime="2026-10-30 07:25:06.596418481 +0000 UTC m=+1667.029182371"
Oct 30 07:25:06.945511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.945475    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 07:25:06.950013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.949987    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962276    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962404    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962486    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962537    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962658    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962802    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962856    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbh9\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.962972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962951    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.963024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.962997    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:06.982350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:06.968730    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-14271d-openshift-default-dockercfg-z2jwj\""
Oct 30 07:25:07.005365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.005329    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 07:25:07.020199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.020169    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:07.084101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.084059    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.114591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.114551    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.115333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.115217    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.122406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.086773    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.122598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.115399    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbh9\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.122801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.122782    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.122910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.122897    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.123044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.123031    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.123163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.123151    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.123278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.123263    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.156789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.128986    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.180400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.129341    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.180794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.141057    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.180956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.149694    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.181067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.153172    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.181192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.178012    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbh9\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.181403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.178734    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.194928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.194853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\") pod \"e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") " pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.302835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.302793    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:07.303001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.302847    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:07.325448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.325408    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:07.522979 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:25:07.522940    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:25:07.563436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.562327    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:07.609203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.609165    6738 generic.go:356] "Generic (PLEG): container finished" podID="f33380a5-dac3-4efb-90de-7de883744c04" containerID="68db72e00ae6b8215b3294e8db9eaff675addcd62d1c4f881d9c21c6f0a68ae7" exitCode=0
Oct 30 07:25:07.609393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.609282    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6" event={"ID":"f33380a5-dac3-4efb-90de-7de883744c04","Type":"ContainerDied","Data":"68db72e00ae6b8215b3294e8db9eaff675addcd62d1c4f881d9c21c6f0a68ae7"}
Oct 30 07:25:07.698370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.697702    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:07.752313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.749767    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 07:25:07.796824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.791715    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:07.796824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.791903    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:07.796824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:07.791968    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:08.302854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.302808    6738 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 07:25:08.303061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.302885    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 07:25:08.589498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.589402    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:08.620611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.620572    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerStarted","Data":"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"}
Oct 30 07:25:08.620925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.620902    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerStarted","Data":"19af91c1d783b0cda6719bd09679420488ae3706237c4e75359618ff35f58160"}
Oct 30 07:25:08.672120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.672053    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podStartSLOduration=2.672031602 podStartE2EDuration="2.672031602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:08.667879745 +0000 UTC m=+1669.100643641" watchObservedRunningTime="2026-10-30 07:25:08.672031602 +0000 UTC m=+1669.104795497"
Oct 30 07:25:08.777898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.777864    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:08.829543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.824741    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\") pod \"f33380a5-dac3-4efb-90de-7de883744c04\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") "
Oct 30 07:25:08.829543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.824846    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\" (UniqueName: \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\") pod \"f33380a5-dac3-4efb-90de-7de883744c04\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") "
Oct 30 07:25:08.829543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.824878    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\") pod \"f33380a5-dac3-4efb-90de-7de883744c04\" (UID: \"f33380a5-dac3-4efb-90de-7de883744c04\") "
Oct 30 07:25:08.829543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.825994    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca" pod "f33380a5-dac3-4efb-90de-7de883744c04" (UID: "f33380a5-dac3-4efb-90de-7de883744c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:08.829933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.829787    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t" pod "f33380a5-dac3-4efb-90de-7de883744c04" (UID: "f33380a5-dac3-4efb-90de-7de883744c04"). InnerVolumeSpecName "kube-api-access-wln5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:08.833493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.832262    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca" pod "f33380a5-dac3-4efb-90de-7de883744c04" (UID: "f33380a5-dac3-4efb-90de-7de883744c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:08.927305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.927265    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:08.927525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.927341    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wln5t\" (UniqueName: \"kubernetes.io/projected/f33380a5-dac3-4efb-90de-7de883744c04-kube-api-access-wln5t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:08.927525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:08.927352    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33380a5-dac3-4efb-90de-7de883744c04-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:09.013489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.013448    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:25:09.014018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.014001    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f33380a5-dac3-4efb-90de-7de883744c04" containerName="revoke-keys"
Oct 30 07:25:09.014018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.014018    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33380a5-dac3-4efb-90de-7de883744c04" containerName="revoke-keys"
Oct 30 07:25:09.014206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.014113    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33380a5-dac3-4efb-90de-7de883744c04" containerName="revoke-keys"
Oct 30 07:25:09.019393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.019362    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.038989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.038958    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"]
Oct 30 07:25:09.044201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.044171    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.052166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.052136    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-265384-openshift-default-dockercfg-tcbq7\""
Oct 30 07:25:09.073916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.073847    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:25:09.075095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.075058    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"]
Oct 30 07:25:09.104862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.104754    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158565    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158612    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2v2\" (UniqueName: \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158705    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158737    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158828    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158920    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158963    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.158989    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.159019    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzv2r\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.159057    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.159098    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.188325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.159128    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263081    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263406    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263561    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263685    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzv2r\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263823    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.263956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.264083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.264235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264220    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.264354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264336    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2v2\" (UniqueName: \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.264530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264515    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.264636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264621    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.264830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.264812    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.265866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.265822    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.265866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263700    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.266264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.266232    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.266826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.263969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.266826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.266642    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.266994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.266919    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.278316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.270129    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.278316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.270920    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.282812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.282507    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.301376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.301280    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzv2r\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.304603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.304564    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.304790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.304639    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2v2\" (UniqueName: \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\") pod \"maas-api-revoke-sub-57e2cac9-582xr\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.325634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.325598    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:09.331438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.331408    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:09.343266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.343215    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:09.363770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.363690    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:09.631442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.631404    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"
Oct 30 07:25:09.631442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.631419    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6" event={"ID":"f33380a5-dac3-4efb-90de-7de883744c04","Type":"ContainerDied","Data":"c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926"}
Oct 30 07:25:09.631897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.631472    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f93981290c1d558647d9d30605e2794c55d22f35d11f50d462159377c42926"
Oct 30 07:25:09.631978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.631907    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:09.633085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.633063    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:25:09.649453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.649423    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:09.728154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.728112    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:09.728282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.728206    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:09.728282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.728249    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:09.731924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.731897    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"]
Oct 30 07:25:09.738908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.736244    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:25:09.775875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.774557    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:09.775875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.775692    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:09.776641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.776616    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:25:09.851040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:09.851004    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:10.644964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.644919    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr" event={"ID":"5a33b917-2d25-4698-a379-92306d95c6bb","Type":"ContainerStarted","Data":"18e33d7c4e923038a92678beb779931642b74d54a3e3fdd3698f586b615bc837"}
Oct 30 07:25:10.645602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.645025    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr" event={"ID":"5a33b917-2d25-4698-a379-92306d95c6bb","Type":"ContainerStarted","Data":"cd9bd1687f24078fc52d493eff17fdec974ee98c86ac8de2586417a84f4f89d3"}
Oct 30 07:25:10.648093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.648058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerStarted","Data":"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"}
Oct 30 07:25:10.648252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.648116    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerStarted","Data":"c8fdba433d3ca151323ac9df65681bda45413071210426b9be28cca54ede6d7c"}
Oct 30 07:25:10.687210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.687157    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr" podStartSLOduration=2.6871450279999998 podStartE2EDuration="2.687145028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:10.686879683 +0000 UTC m=+1671.119643576" watchObservedRunningTime="2026-10-30 07:25:10.687145028 +0000 UTC m=+1671.119908920"
Oct 30 07:25:10.733457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:10.733398    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podStartSLOduration=2.733379893 podStartE2EDuration="2.733379893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:10.731912973 +0000 UTC m=+1671.164676880" watchObservedRunningTime="2026-10-30 07:25:10.733379893 +0000 UTC m=+1671.166143789"
Oct 30 07:25:11.343488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.343449    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:11.349694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.349666    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:11.506870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.506837    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:11.657409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.657370    6738 generic.go:356] "Generic (PLEG): container finished" podID="5a33b917-2d25-4698-a379-92306d95c6bb" containerID="18e33d7c4e923038a92678beb779931642b74d54a3e3fdd3698f586b615bc837" exitCode=0
Oct 30 07:25:11.657826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.657457    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr" event={"ID":"5a33b917-2d25-4698-a379-92306d95c6bb","Type":"ContainerDied","Data":"18e33d7c4e923038a92678beb779931642b74d54a3e3fdd3698f586b615bc837"}
Oct 30 07:25:11.658068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.658048    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:11.658861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.658843    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:25:11.676695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:11.676651    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:12.029624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.029535    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:25:12.034105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.034081    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.062324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.062278    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:25:12.111137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.111094    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.111407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.111197    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.129783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129429    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.129783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129516    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkt8\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.129783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129598    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.129783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129664    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.129783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129768    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.130258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129881    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.130258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.129928    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.155942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.155901    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:12.230937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.230898    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.230937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.230940    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkt8\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231069    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231181    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231198    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231229    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231368    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231408    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231516    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231651    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.231745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231692    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.232000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.231974    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.233527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.233505    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.233905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.233886    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.256695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.256664    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.259609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.259581    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkt8\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.348682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.348586    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:12.674470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.674433    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:25:12.676289 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:12.676253    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f2806b_eef5_4ce8_b7ee_93c5605871b3.slice/crio-dbe8f3b61ee96f38d1317f8e52bcb9f5e75b5689b8d7cfd4a80a336b82b0d21c WatchSource:0}: Error finding container dbe8f3b61ee96f38d1317f8e52bcb9f5e75b5689b8d7cfd4a80a336b82b0d21c: Status 404 returned error can't find the container with id dbe8f3b61ee96f38d1317f8e52bcb9f5e75b5689b8d7cfd4a80a336b82b0d21c
Oct 30 07:25:12.679219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.679185    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:12.679371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.679258    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:12.679371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.679289    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:12.803902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.803869    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:12.941138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.941053    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\") pod \"5a33b917-2d25-4698-a379-92306d95c6bb\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") "
Oct 30 07:25:12.941314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.941260    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\") pod \"5a33b917-2d25-4698-a379-92306d95c6bb\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") "
Oct 30 07:25:12.941388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.941353    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\" (UniqueName: \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\") pod \"5a33b917-2d25-4698-a379-92306d95c6bb\" (UID: \"5a33b917-2d25-4698-a379-92306d95c6bb\") "
Oct 30 07:25:12.941694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.941661    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca" pod "5a33b917-2d25-4698-a379-92306d95c6bb" (UID: "5a33b917-2d25-4698-a379-92306d95c6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:12.942050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.942028    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:12.943369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.943347    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca" pod "5a33b917-2d25-4698-a379-92306d95c6bb" (UID: "5a33b917-2d25-4698-a379-92306d95c6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:12.943453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:12.943386    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2" pod "5a33b917-2d25-4698-a379-92306d95c6bb" (UID: "5a33b917-2d25-4698-a379-92306d95c6bb"). InnerVolumeSpecName "kube-api-access-gn2v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:13.043922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.043887    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a33b917-2d25-4698-a379-92306d95c6bb-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:13.043922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.043923    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2v2\" (UniqueName: \"kubernetes.io/projected/5a33b917-2d25-4698-a379-92306d95c6bb-kube-api-access-gn2v2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:13.103382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.103345    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:13.222847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.222758    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:13.269460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.269417    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 07:25:13.270053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.270034    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5a33b917-2d25-4698-a379-92306d95c6bb" containerName="revoke-keys"
Oct 30 07:25:13.270150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.270055    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a33b917-2d25-4698-a379-92306d95c6bb" containerName="revoke-keys"
Oct 30 07:25:13.270206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.270190    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a33b917-2d25-4698-a379-92306d95c6bb" containerName="revoke-keys"
Oct 30 07:25:13.276306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.275539    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.291874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.291845    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 07:25:13.292137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.292114    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-14271d\""
Oct 30 07:25:13.292137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.292113    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-14271d-dockercfg-kt4vb\""
Oct 30 07:25:13.367329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.361693    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.368256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.368209    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k7q\" (UniqueName: \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.485369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.483099    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:13.491856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.491378    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 07:25:13.502061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.496013    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k7q\" (UniqueName: \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.502061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.496206    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.502061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.497152    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.510761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.510734    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 07:25:13.510909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.510819    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.522216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.522186    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k7q\" (UniqueName: \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\") pod \"payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.594269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.594237    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:13.597657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.597473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p478v\" (UniqueName: \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.597657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.597536    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.602731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.602704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:13.708973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.708888    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"
Oct 30 07:25:13.711561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.709963    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p478v\" (UniqueName: \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.711561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.710063    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.711561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.711102    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.721720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.721687    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" event={"ID":"41f2806b-eef5-4ce8-b7ee-93c5605871b3","Type":"ContainerStarted","Data":"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"}
Oct 30 07:25:13.721858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.721735    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" event={"ID":"41f2806b-eef5-4ce8-b7ee-93c5605871b3","Type":"ContainerStarted","Data":"dbe8f3b61ee96f38d1317f8e52bcb9f5e75b5689b8d7cfd4a80a336b82b0d21c"}
Oct 30 07:25:13.721858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.721753    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr" event={"ID":"5a33b917-2d25-4698-a379-92306d95c6bb","Type":"ContainerDied","Data":"cd9bd1687f24078fc52d493eff17fdec974ee98c86ac8de2586417a84f4f89d3"}
Oct 30 07:25:13.721858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.721771    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd9bd1687f24078fc52d493eff17fdec974ee98c86ac8de2586417a84f4f89d3"
Oct 30 07:25:13.738370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.736553    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p478v\" (UniqueName: \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\") pod \"payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.749646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.749540    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podStartSLOduration=2.749523008 podStartE2EDuration="2.749523008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:13.745955508 +0000 UTC m=+1674.178719400" watchObservedRunningTime="2026-10-30 07:25:13.749523008 +0000 UTC m=+1674.182286901"
Oct 30 07:25:13.826783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.826740    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:13.962719 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:13.962654    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1d3bca_6e3b_448f_a8fc_48e635265401.slice/crio-67df1517a961539e62ab87950e2031d98ee7d9e7b765855591251f5d910f35bb WatchSource:0}: Error finding container 67df1517a961539e62ab87950e2031d98ee7d9e7b765855591251f5d910f35bb: Status 404 returned error can't find the container with id 67df1517a961539e62ab87950e2031d98ee7d9e7b765855591251f5d910f35bb
Oct 30 07:25:13.963791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:13.963766    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 07:25:14.157272 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:14.157236    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3d9883_65cb_4fd5_aa1d_cf514f1ab763.slice/crio-88b4fd44247d9bf53572a72b6288b6134be0b13639a543880adbe9cbb93ab72a WatchSource:0}: Error finding container 88b4fd44247d9bf53572a72b6288b6134be0b13639a543880adbe9cbb93ab72a: Status 404 returned error can't find the container with id 88b4fd44247d9bf53572a72b6288b6134be0b13639a543880adbe9cbb93ab72a
Oct 30 07:25:14.157419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.157339    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 07:25:14.349137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.349096    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:14.350666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.350641    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:14.350805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.350690    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:14.658728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.658677    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:14.720946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.720911    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" event={"ID":"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763","Type":"ContainerStarted","Data":"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"}
Oct 30 07:25:14.721471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.720983    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" event={"ID":"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763","Type":"ContainerStarted","Data":"88b4fd44247d9bf53572a72b6288b6134be0b13639a543880adbe9cbb93ab72a"}
Oct 30 07:25:14.721471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.721062    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:14.722619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.722494    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" event={"ID":"ac1d3bca-6e3b-448f-a8fc-48e635265401","Type":"ContainerStarted","Data":"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"}
Oct 30 07:25:14.722619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.722552    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" event={"ID":"ac1d3bca-6e3b-448f-a8fc-48e635265401","Type":"ContainerStarted","Data":"67df1517a961539e62ab87950e2031d98ee7d9e7b765855591251f5d910f35bb"}
Oct 30 07:25:14.722619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.722580    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:14.762843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.762743    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" podStartSLOduration=1.762726789 podStartE2EDuration="1.762726789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:14.761202883 +0000 UTC m=+1675.193966777" watchObservedRunningTime="2026-10-30 07:25:14.762726789 +0000 UTC m=+1675.195490682"
Oct 30 07:25:14.808668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:14.808617    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" podStartSLOduration=1.808602569 podStartE2EDuration="1.808602569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:14.806856982 +0000 UTC m=+1675.239620923" watchObservedRunningTime="2026-10-30 07:25:14.808602569 +0000 UTC m=+1675.241366462"
Oct 30 07:25:15.337675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.337633    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:15.349803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.349766    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:15.349971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.349819    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:15.466331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.466272    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:15.494901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.490916    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 07:25:15.496244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.496209    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.502758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.501958    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 07:25:15.511575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.511548    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-265384\""
Oct 30 07:25:15.511702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.511688    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-265384-dockercfg-cpq2f\""
Oct 30 07:25:15.562415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.545353    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.562415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.545528    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdwd\" (UniqueName: \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.676581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.656113    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdwd\" (UniqueName: \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.676581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.656382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.676581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.657715    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.753613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.753582    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 07:25:15.756137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.756102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:15.757844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.757809    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 07:25:15.757970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.757914    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.758660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.758634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdwd\" (UniqueName: \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\") pod \"payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.813116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.813083    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:15.860753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.860696    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.860994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.860975    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ptc\" (UniqueName: \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.962996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.962905    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.963179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.963162    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ptc\" (UniqueName: \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.963725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.963704    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:15.995815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:15.995732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ptc\" (UniqueName: \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\") pod \"payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:16.120979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.120947    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:16.154282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.154241    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 07:25:16.160168 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:16.159877    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359e565c_47a4_4762_9da7_00c983cb06b2.slice/crio-0ac94ef8d0baa27dd451cca96c0c7053b19617e534f3dab6846da70ad978d862 WatchSource:0}: Error finding container 0ac94ef8d0baa27dd451cca96c0c7053b19617e534f3dab6846da70ad978d862: Status 404 returned error can't find the container with id 0ac94ef8d0baa27dd451cca96c0c7053b19617e534f3dab6846da70ad978d862
Oct 30 07:25:16.160592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.160565    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 07:25:16.160708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.160692    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.212380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.208561    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-932c00-openshift-default-dockercfg-bdh8l\""
Oct 30 07:25:16.212380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.209697    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:16.212724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.212652    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 07:25:16.358316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.357865    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:16.358316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.357945    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.381857    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mk5\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.381915    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382004    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382092    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382131    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382165    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382217    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382339    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.383319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.382371    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.429456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.395591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483227    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483353    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483446    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483485    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483516    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483567    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483656    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483686    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.483776    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mk5\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.484660    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.484938    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.487931    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.488187    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.494377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.488525    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.495314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.494788    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:25:16.556359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.550360    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.556359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.550245    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:25:16.557025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.556985    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.557163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.557125    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:25:16.557229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.557164    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:25:16.557286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.557257    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.557478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.557448    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.557668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.557643    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.560284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.560130    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mk5\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\") pod \"e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") " pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.566396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.566363    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe2ed-openshift-default-dockercfg-s6hbq\""
Oct 30 07:25:16.689431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.687771    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691345    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgbc\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691467    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691543    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691572    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691721    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqpf\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691833    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691865    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691934    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.691993    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692018    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692131    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692158    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692222    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692283    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692425    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692455    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692523    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.695954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.692582    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.794241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.794188    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgbc\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.794829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.794800    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.794941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.794853    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.794941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.794878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.794991    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqpf\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795087    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795160    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795190    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795285    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795330    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795362    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795415    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795528    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.795664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795554    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795588    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.795664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.795617    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.796245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.796033    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.796396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.796334    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.796665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.796646    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.796935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.796912    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.797177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.797017    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.798124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.797578    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.798124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.797825    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.798124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.798068    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.798124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.798088    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.798431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.798262    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.799569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.799537    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.799971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.799944    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.800649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.800617    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.802235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.802209    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.805473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.805447    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" event={"ID":"85041ce5-3e68-4edc-b5d2-9e0a3e880a83","Type":"ContainerStarted","Data":"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"}
Oct 30 07:25:16.805579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.805495    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" event={"ID":"85041ce5-3e68-4edc-b5d2-9e0a3e880a83","Type":"ContainerStarted","Data":"f550d5c88132fac8512e4f2e8ed6e41aeef18733c9f6b04bde9a6b294d035f2b"}
Oct 30 07:25:16.805579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.805546    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:16.807563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.807539    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" event={"ID":"359e565c-47a4-4762-9da7-00c983cb06b2","Type":"ContainerStarted","Data":"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"}
Oct 30 07:25:16.807657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.807593    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" event={"ID":"359e565c-47a4-4762-9da7-00c983cb06b2","Type":"ContainerStarted","Data":"0ac94ef8d0baa27dd451cca96c0c7053b19617e534f3dab6846da70ad978d862"}
Oct 30 07:25:16.807719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.807675    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:16.817759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.817727    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgbc\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.819272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.819245    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.819499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.819483    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqpf\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.820069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.820052    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.843402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.843376    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:16.848524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.848410    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" podStartSLOduration=1.848395826 podStartE2EDuration="1.848395826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:16.845169984 +0000 UTC m=+1677.277933876" watchObservedRunningTime="2026-10-30 07:25:16.848395826 +0000 UTC m=+1677.281159719"
Oct 30 07:25:16.894453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.894157    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:16.895544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.895483    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" podStartSLOduration=1.895466302 podStartE2EDuration="1.895466302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:16.893916336 +0000 UTC m=+1677.326680230" watchObservedRunningTime="2026-10-30 07:25:16.895466302 +0000 UTC m=+1677.328230194"
Oct 30 07:25:16.910216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.910189    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:16.941745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:16.940726    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:17.215486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.215455    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 07:25:17.217022 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:17.216993    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdaeb338_56ba_43c4_aaee_36edb3fa3862.slice/crio-abf73941de5bba4f67ae679e0af830d7f35da7f4d6ecb1a97c9f93323d0a39e1 WatchSource:0}: Error finding container abf73941de5bba4f67ae679e0af830d7f35da7f4d6ecb1a97c9f93323d0a39e1: Status 404 returned error can't find the container with id abf73941de5bba4f67ae679e0af830d7f35da7f4d6ecb1a97c9f93323d0a39e1
Oct 30 07:25:17.220129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.220088    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.220224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.220178    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.220264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.220222    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.262158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.262131    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:25:17.262457 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:17.262435    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a0f1ce_51ab_4fec_9879_5717948d191f.slice/crio-fcfd3292aa0865ffbafaf01ad6ea69454002599152537734c0a69c0ba0760a08 WatchSource:0}: Error finding container fcfd3292aa0865ffbafaf01ad6ea69454002599152537734c0a69c0ba0760a08: Status 404 returned error can't find the container with id fcfd3292aa0865ffbafaf01ad6ea69454002599152537734c0a69c0ba0760a08
Oct 30 07:25:17.265704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.265659    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.265811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.265789    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.265868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.265843    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.335012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.334983    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:25:17.348278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.348233    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.348447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.348344    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.348447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.348391    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:17.349036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.348998    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:17.349141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.349049    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:17.359350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.359286    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"]
Oct 30 07:25:17.365635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.365614    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.400830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.395519    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:25:17.412508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.411766    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:25:17.425970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.425901    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:25:17.425970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.425943    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:25:17.427904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.427874    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"]
Oct 30 07:25:17.515284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.515248    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.515518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.515358    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7x4\" (UniqueName: \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.515518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.515444    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.617851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.617814    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.618080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.618062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.618168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.618117    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7x4\" (UniqueName: \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.619588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.619562    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.622051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.622012    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.642312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.642252    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7x4\" (UniqueName: \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\") pod \"maas-api-revoke-sub-c51e3423-klr6x\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.749954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.749917    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:17.830824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.830767    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerStarted","Data":"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"}
Oct 30 07:25:17.831286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.830868    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerStarted","Data":"abf73941de5bba4f67ae679e0af830d7f35da7f4d6ecb1a97c9f93323d0a39e1"}
Oct 30 07:25:17.834279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.834243    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" event={"ID":"a812e3c0-93d1-45ad-9df8-6467858de0f1","Type":"ContainerStarted","Data":"5f0410dff540774ec3fdbe8cc8570fbe31822f831e1101da80bac2b3f1074532"}
Oct 30 07:25:17.834630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.834607    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" event={"ID":"a812e3c0-93d1-45ad-9df8-6467858de0f1","Type":"ContainerStarted","Data":"0e94b660b720368e612d0fe12b56aea98a071f4fe256c37c23377e0a2565062c"}
Oct 30 07:25:17.838194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.838057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" event={"ID":"f7a0f1ce-51ab-4fec-9879-5717948d191f","Type":"ContainerStarted","Data":"4f31bfb861fed986959a2f28426f599fc9d219fddb466668ecd9f0df022614ac"}
Oct 30 07:25:17.838194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.838118    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" event={"ID":"f7a0f1ce-51ab-4fec-9879-5717948d191f","Type":"ContainerStarted","Data":"fcfd3292aa0865ffbafaf01ad6ea69454002599152537734c0a69c0ba0760a08"}
Oct 30 07:25:17.844525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.844487    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:17.907518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.907424    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:17.913433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.907720    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podStartSLOduration=1.907699369 podStartE2EDuration="1.907699369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:17.890644255 +0000 UTC m=+1678.323408149" watchObservedRunningTime="2026-10-30 07:25:17.907699369 +0000 UTC m=+1678.340463262"
Oct 30 07:25:17.913979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.913946    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:17.943676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:17.943596    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podStartSLOduration=1.943574082 podStartE2EDuration="1.943574082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:17.939509833 +0000 UTC m=+1678.372273756" watchObservedRunningTime="2026-10-30 07:25:17.943574082 +0000 UTC m=+1678.376337975"
Oct 30 07:25:18.248189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.190281    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podStartSLOduration=2.190259397 podStartE2EDuration="2.190259397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:18.066439531 +0000 UTC m=+1678.499203428" watchObservedRunningTime="2026-10-30 07:25:18.190259397 +0000 UTC m=+1678.623023290"
Oct 30 07:25:18.256486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.256434    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"]
Oct 30 07:25:18.360036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.359995    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:18.360244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.360055    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:18.877230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.877194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x" event={"ID":"b711cbd8-08b9-4b77-a145-adc6e8349311","Type":"ContainerStarted","Data":"b7dcf79e8e64569ba78c6c4f1727bb5aa27bdb298a4c3084a8b7b339dd18ce37"}
Oct 30 07:25:18.881565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.881537    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:18.928137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.928076    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:18.928533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.928500    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:18.929125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.929103    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:18.970112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:18.970066    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:19.355937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.355850    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:19.356254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.356226    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:19.571540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.571489    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:19.750353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.711913    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:25:19.750353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.746641    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:19.773448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.773405    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:19.916321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.896679    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:19.919766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.919718    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:19.920406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.912036    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:19.920591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.920568    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:19.995383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:19.994392    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:20.157885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.157839    6738 generic.go:356] "Generic (PLEG): container finished" podID="b711cbd8-08b9-4b77-a145-adc6e8349311" containerID="8ce921f165e52b3b3b6f6e58ed3f5e1606816cb1d2fe10114ca5d69d5856e903" exitCode=0
Oct 30 07:25:20.174321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.162246    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x" event={"ID":"b711cbd8-08b9-4b77-a145-adc6e8349311","Type":"ContainerDied","Data":"8ce921f165e52b3b3b6f6e58ed3f5e1606816cb1d2fe10114ca5d69d5856e903"}
Oct 30 07:25:20.182793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.182753    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:20.204319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.202779    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:20.231320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.205822    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:25:20.267399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.257342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:20.389327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.378432    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:20.402325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.389630    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:20.576018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.574732    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:20.576018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.575722    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:25:20.796320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.741373    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:25:20.796320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.742109    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:25:20.796320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.791645    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:20.845370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.802604    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:25:20.852321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:20.809590    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.179877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.141439    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 07:25:21.179877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.146218    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:25:21.226320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.222707    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.270604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.141711    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:25:21.270604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.263174    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:21.270604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.263318    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:21.270604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.263399    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:21.270604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.263425    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:21.394355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.303931    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.394355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.385749    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:25:21.394355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.385900    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:25:21.545477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.545369    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.858569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.602129    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.858569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.832485    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.858569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:21.630052    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:22.043121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.043085    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.043430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.043395    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:22.051192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.044420    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.096089    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.098592    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.135608    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.169736    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.171218    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.171263    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.173593    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.184320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.173858    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 07:25:22.275319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.258155    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:25:22.284564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.284516    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:25:22.286623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.286588    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.384403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.384369    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.384682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.384654    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 07:25:22.642511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.642458    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:22.941378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.932887    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.941378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.932946    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:22.975320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:22.946054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:23.008277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.005218    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:23.008277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.005287    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:23.008277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.007149    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.231687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.188179    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:23.234833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.234784    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.236155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.236122    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:23.240192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.240160    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.249967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.249928    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.268331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.264976    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:25:23.293461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.293413    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.306462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.306418    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.951579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:23.951530    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.012178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.009857    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:24.012178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.009915    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:24.029146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.029108    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:24.029467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.029421    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:24.045774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.045739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:24.051849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.051802    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.059906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.059853    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 07:25:24.240229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.240193    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:24.240909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.240880    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.297395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.278438    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.301410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.301239    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:25:24.324007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.321576    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.324007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.322118    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:24.384782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.324908    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.384782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.365986    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:24.452316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.451305    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\") pod \"b711cbd8-08b9-4b77-a145-adc6e8349311\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") "
Oct 30 07:25:24.452316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.451443    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\" (UniqueName: \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\") pod \"b711cbd8-08b9-4b77-a145-adc6e8349311\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") "
Oct 30 07:25:24.452316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.451592    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\") pod \"b711cbd8-08b9-4b77-a145-adc6e8349311\" (UID: \"b711cbd8-08b9-4b77-a145-adc6e8349311\") "
Oct 30 07:25:24.479768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.462060    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:25:24.479768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.462110    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 07:25:24.479768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.463072    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b711cbd8-08b9-4b77-a145-adc6e8349311" containerName="revoke-keys"
Oct 30 07:25:24.479768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.463095    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b711cbd8-08b9-4b77-a145-adc6e8349311" containerName="revoke-keys"
Oct 30 07:25:24.479768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.463274    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b711cbd8-08b9-4b77-a145-adc6e8349311" containerName="revoke-keys"
Oct 30 07:25:24.488073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.480988    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 07:25:24.488073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.481101    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:24.488073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.481395    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.512327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.503413    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-932c00-dockercfg-7p95s\""
Oct 30 07:25:24.512327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.510259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:24.525357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.512602    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:25:24.563626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.548199    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-932c00\""
Oct 30 07:25:24.566720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.564177    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm95\" (UniqueName: \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.566720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.565121    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.594632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.594560    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca" pod "b711cbd8-08b9-4b77-a145-adc6e8349311" (UID: "b711cbd8-08b9-4b77-a145-adc6e8349311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:24.715315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.714793    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:25:24.858061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.857949    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm95\" (UniqueName: \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.866522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.866474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.866778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.866759    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:24.890344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.867781    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.963171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.963142    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:25:24.963835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.963812    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:25:24.963980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.963907    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:25:24.964057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.963980    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:25:24.964057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.964054    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:25:24.964209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.963218    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm95\" (UniqueName: \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\") pod \"payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:24.964347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.860390    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4" pod "b711cbd8-08b9-4b77-a145-adc6e8349311" (UID: "b711cbd8-08b9-4b77-a145-adc6e8349311"). InnerVolumeSpecName "kube-api-access-9g7x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:24.964347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.864963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca" pod "b711cbd8-08b9-4b77-a145-adc6e8349311" (UID: "b711cbd8-08b9-4b77-a145-adc6e8349311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:24.964505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.964481    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:24.964583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.964539    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:24.965822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.965798    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:24.965942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.965852    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:24.971325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:24.967829    6738 scope.go:122] "RemoveContainer" containerID="790756428d8ad16ad6b47e18a177fc81ff3a52be1269ac82f2c9e21f380c18bb"
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.044775    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.044966    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x" event={"ID":"b711cbd8-08b9-4b77-a145-adc6e8349311","Type":"ContainerDied","Data":"b7dcf79e8e64569ba78c6c4f1727bb5aa27bdb298a4c3084a8b7b339dd18ce37"}
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.045021    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7dcf79e8e64569ba78c6c4f1727bb5aa27bdb298a4c3084a8b7b339dd18ce37"
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.047871    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.050075    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7x4\" (UniqueName: \"kubernetes.io/projected/b711cbd8-08b9-4b77-a145-adc6e8349311-kube-api-access-9g7x4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:25.052568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.051628    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b711cbd8-08b9-4b77-a145-adc6e8349311-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:25:25.160011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.159982    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:25:25.264890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.264859    6738 scope.go:122] "RemoveContainer" containerID="068b3ff63204f4ff26c060133f7aa408f14bcd43477d63baf6f459b8da440438"
Oct 30 07:25:25.440643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.411482    6738 scope.go:122] "RemoveContainer" containerID="cdd17a97c7789bd1d719f11bc9bbbe83487cd29b61f29bef6667d0a4d71578c1"
Oct 30 07:25:25.706561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.706469    6738 scope.go:122] "RemoveContainer" containerID="4358e9d3ff842b48dd50c44ded73b99a5ab9086b1ade102c9733b5bb5552c629"
Oct 30 07:25:25.760760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.758282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:25.862048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.862019    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 07:25:25.897595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.897560    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:25.898357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.898324    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:25.935078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.935033    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:25.935395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:25.935360    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:26.436363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.432973    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 07:25:26.438660 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:26.438630    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b181a4_57f2_4897_a695_4aaf837e1647.slice/crio-98ed1235984a9868afe0cb3ab7ae412085d622cacd0c032ff6e3264c60fee750 WatchSource:0}: Error finding container 98ed1235984a9868afe0cb3ab7ae412085d622cacd0c032ff6e3264c60fee750: Status 404 returned error can't find the container with id 98ed1235984a9868afe0cb3ab7ae412085d622cacd0c032ff6e3264c60fee750
Oct 30 07:25:26.822982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.817796    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:26.822982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.819552    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 07:25:26.902457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.899392    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:26.902457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.899451    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:26.921506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.921457    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:26.921830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.921781    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:26.955707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:26.955667    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:27.229974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.229934    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" event={"ID":"e7b181a4-57f2-4897-a695-4aaf837e1647","Type":"ContainerStarted","Data":"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"}
Oct 30 07:25:27.230198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.229998    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:27.230198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.230015    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" event={"ID":"e7b181a4-57f2-4897-a695-4aaf837e1647","Type":"ContainerStarted","Data":"98ed1235984a9868afe0cb3ab7ae412085d622cacd0c032ff6e3264c60fee750"}
Oct 30 07:25:27.842064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.842026    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 07:25:27.842616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.842117    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 07:25:27.895169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.895055    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 07:25:27.895169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.895114    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 07:25:27.897823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.897763    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" podStartSLOduration=4.897743839 podStartE2EDuration="4.897743839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:27.277452816 +0000 UTC m=+1687.710216732" watchObservedRunningTime="2026-10-30 07:25:27.897743839 +0000 UTC m=+1688.330507736"
Oct 30 07:25:27.911440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.911404    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 07:25:27.911646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:27.911464    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 07:25:28.193869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.193835    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:28.327319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.324815    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:28.648463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.648398    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:28.797355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.797280    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:28.818954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.818898    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 07:25:28.827761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.827721    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:28.852840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.851783    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 07:25:28.963374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:28.953848    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bd8589-openshift-default-dockercfg-72qhh\""
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037624    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckt2m\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037792    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037860    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037907    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.037973    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.038038    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.038135    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.048318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.038180    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.133319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.132445    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:29.148323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.142097    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.148323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.142168    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.148323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.142280    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckt2m\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.166337    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.166211    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.166599    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.166940    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.167087    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.167189    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.167326    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.167493    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.168773    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.169340    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.180077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.173423    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.190371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.185256    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckt2m\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.194994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.194857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.210715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.210671    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.214149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.214035    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.279030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.278987    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:29.468927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.468834    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:29.582929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.563034    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:29.582929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.564593    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:29.583177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.583011    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:25:29.820313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.820265    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 07:25:29.820832 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:29.820789    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183fa159_dc30_4e2d_953e_e83ed0986556.slice/crio-64e48dc94bdfb1a2ebb4d7746a8c7ac87f4b5c9758b8cf1a0f482402ad0149f9 WatchSource:0}: Error finding container 64e48dc94bdfb1a2ebb4d7746a8c7ac87f4b5c9758b8cf1a0f482402ad0149f9: Status 404 returned error can't find the container with id 64e48dc94bdfb1a2ebb4d7746a8c7ac87f4b5c9758b8cf1a0f482402ad0149f9
Oct 30 07:25:29.823845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.823802    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:29.823961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.823921    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:29.824028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.823970    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:29.948865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.948432    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:29.948865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:29.948482    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:30.376341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.366612    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerStarted","Data":"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"}
Oct 30 07:25:30.376341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.366673    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerStarted","Data":"64e48dc94bdfb1a2ebb4d7746a8c7ac87f4b5c9758b8cf1a0f482402ad0149f9"}
Oct 30 07:25:30.441121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.441036    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podStartSLOduration=2.441018633 podStartE2EDuration="2.441018633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:30.440074261 +0000 UTC m=+1690.872838154" watchObservedRunningTime="2026-10-30 07:25:30.441018633 +0000 UTC m=+1690.873782527"
Oct 30 07:25:30.469549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.469446    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:30.477688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.477653    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:30.582220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.581759    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:30.622050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.622012    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:30.622812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.622779    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:30.623951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.623912    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:25:30.698275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.698184    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:30.839039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:30.839005    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:31.378315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.378265    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:31.381869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.381843    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:25:31.513230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.511939    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:25:31.519351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.519316    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.519514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.519467    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:25:31.525351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.525278    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.568165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.568131    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:25:31.573798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.573760    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:25:31.672655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672552    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.672655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672614    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.672864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672736    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.672864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672810    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.672864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672854    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.672976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672878    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.672976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672901    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.672976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.672931    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.673072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673034    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.673110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673068    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.673110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673090    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.673199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673164    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbqd\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.673199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673194    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.673345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673218    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.673345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673269    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.673441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673383    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgl2r\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.673441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673407    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.673441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.673434    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805223    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805282    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805349    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805387    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805422    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805446    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805489    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805528    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805562    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805608    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbqd\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805656    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805694    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805731    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805805    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgl2r\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805838    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.805879    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.806029    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.806079    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.806589    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.807326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.807224    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.808678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.807571    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.808678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.807801    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.813689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.813662    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.814054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.814020    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.814499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.814478    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.814879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.814848    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.815280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.815258    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.816037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.816015    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.852042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.849025    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.852042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.849682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgl2r\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.852319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.852093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.853209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.852987    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.855131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.855092    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.855488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.855465    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:31.862349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.856312    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.862349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.858767    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbqd\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:31.886657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:31.886621    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:32.155929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.155896    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:32.226626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.225758    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:25:32.229305 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:32.229253    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23e079a_adb3_4c16_9450_82ac846243b9.slice/crio-76952dd0262f5df0f45eeca79e7908ab2e84c956cb10dc6a316f4c47b7bc189d WatchSource:0}: Error finding container 76952dd0262f5df0f45eeca79e7908ab2e84c956cb10dc6a316f4c47b7bc189d: Status 404 returned error can't find the container with id 76952dd0262f5df0f45eeca79e7908ab2e84c956cb10dc6a316f4c47b7bc189d
Oct 30 07:25:32.232464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.232412    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.232586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.232501    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.232586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.232549    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.387744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.387703    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" event={"ID":"a23e079a-adb3-4c16-9450-82ac846243b9","Type":"ContainerStarted","Data":"accb17c965516b2a7b73d9c94dd46f78333ae53e667a00960b3f185792473d2d"}
Oct 30 07:25:32.388264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.387769    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" event={"ID":"a23e079a-adb3-4c16-9450-82ac846243b9","Type":"ContainerStarted","Data":"76952dd0262f5df0f45eeca79e7908ab2e84c956cb10dc6a316f4c47b7bc189d"}
Oct 30 07:25:32.513057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.512916    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podStartSLOduration=1.512895873 podStartE2EDuration="1.512895873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:32.480151911 +0000 UTC m=+1692.912915837" watchObservedRunningTime="2026-10-30 07:25:32.512895873 +0000 UTC m=+1692.945659771"
Oct 30 07:25:32.518323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.518273    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:25:32.522673 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:32.522634    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e1ad14_0cbb_4be0_8eaf_0518ffe7596b.slice/crio-025b1d55760b39078d2a8df60b9ab45cdc684fbbfecfe3d53183a50200f6f39a WatchSource:0}: Error finding container 025b1d55760b39078d2a8df60b9ab45cdc684fbbfecfe3d53183a50200f6f39a: Status 404 returned error can't find the container with id 025b1d55760b39078d2a8df60b9ab45cdc684fbbfecfe3d53183a50200f6f39a
Oct 30 07:25:32.525869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.525826    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.526146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.526112    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.526252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.526174    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:32.789032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.788932    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:32.898318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.889401    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:32.907093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:32.907055    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:33.029319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.028869    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:33.247322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.245418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:33.380726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.380663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:33.434694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.434658    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" event={"ID":"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b","Type":"ContainerStarted","Data":"ce35f28a00f1193fd282d942a5aa4ae1256c6b81c63cc2cf168b05eebba71898"}
Oct 30 07:25:33.435179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.434808    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" event={"ID":"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b","Type":"ContainerStarted","Data":"025b1d55760b39078d2a8df60b9ab45cdc684fbbfecfe3d53183a50200f6f39a"}
Oct 30 07:25:33.526378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.511933    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podStartSLOduration=2.511908318 podStartE2EDuration="2.511908318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:33.498805121 +0000 UTC m=+1693.931569013" watchObservedRunningTime="2026-10-30 07:25:33.511908318 +0000 UTC m=+1693.944672212"
Oct 30 07:25:33.529934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.529895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:33.912437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.908436    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:33.912437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:33.908515    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:34.156629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:34.156593    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:34.157645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:34.157624    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:34.157722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:34.157672    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:34.900320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:34.890670    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:34.900320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:34.890746    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:35.123813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.123776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:35.156501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.156411    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:35.156501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.156462    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:35.249143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.249106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:35.321323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.317365    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 07:25:35.332523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.332490    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.334269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.334241    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 07:25:35.438394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.432790    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-bd8589-dockercfg-xkwsn\""
Oct 30 07:25:35.438394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.432915    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-bd8589\""
Oct 30 07:25:35.473321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.472857    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:35.595321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.585797    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.595321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.585965    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mgl\" (UniqueName: \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.713377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.705675    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mgl\" (UniqueName: \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.713377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.705901    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.764317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.761599    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.838666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.838604    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mgl\" (UniqueName: \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\") pod \"payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:35.905813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.905773    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:35.906455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.906402    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:35.950330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:35.950263    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:36.158422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.158385    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:36.158680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.158651    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:36.331025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.330986    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 07:25:36.826947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.826899    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" event={"ID":"1e276cef-464c-4b86-b80c-c26e265f57fb","Type":"ContainerStarted","Data":"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"}
Oct 30 07:25:36.827166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.826965    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" event={"ID":"1e276cef-464c-4b86-b80c-c26e265f57fb","Type":"ContainerStarted","Data":"b4364c27bb96ae88bfc760f0963e477200deb38ad25eea6d08b56b762ce00768"}
Oct 30 07:25:36.827166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.827001    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:36.871319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.868845    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" podStartSLOduration=1.8688244680000001 podStartE2EDuration="1.868824468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:36.866591432 +0000 UTC m=+1697.299355326" watchObservedRunningTime="2026-10-30 07:25:36.868824468 +0000 UTC m=+1697.301588362"
Oct 30 07:25:36.893318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.890698    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:36.893318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:36.890759    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:37.157333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:37.156914    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:37.157333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:37.156975    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:37.544804 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:25:37.544698    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:25:37.894760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:37.894724    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:37.895059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:37.895031    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:37.922103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:37.922064    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:38.156252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:38.156176    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:38.156483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:38.156462    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:38.309318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:38.305507    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 07:25:38.888318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:38.887702    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 07:25:38.888318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:38.887774    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 07:25:39.157350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:39.157226    6738 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 07:25:39.157350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:39.157283    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 07:25:40.059323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.059270    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:40.344841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.344745    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:40.345577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.345536    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:40.346397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.346371    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:25:40.592678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.592648    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:40.593105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.593075    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:40.594094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:40.594070    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:25:42.112715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:42.110994    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:42.232533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:42.232494    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:42.349777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:42.349745    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:42.696734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:42.696693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:44.170331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:44.170271    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:45.937109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:45.937067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:46.048094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.048053    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:46.166356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.166281    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:46.785278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.785227    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:25:46.789957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.789933    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.801119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.801092    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:25:46.806211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.806189    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.816418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.816385    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:25:46.821990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.821960    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-06b26e-openshift-default-dockercfg-t2k6l\""
Oct 30 07:25:46.863889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.863838    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.864277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.864251    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.867278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.867248    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.867574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.867552    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.867868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.867847    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.868056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.868038    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.868234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.868218    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.868427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.868409    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jg4\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.868613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.868594    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.872384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.872352    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:25:46.969906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.969867    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.969925    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.969961    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970000    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970110    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970140    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggwl\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970380    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970428    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970382    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970459    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970499    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970673    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970718    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jg4\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970767    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970812    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.970859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970854    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.971273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970965    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.971273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.970977    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.971273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.971047    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.971273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.971096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:46.971273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.971209    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.971566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.971506    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.972709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.972687    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.972871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.972856    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.994708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.994671    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:46.994835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:46.994782    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jg4\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:47.072397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072285    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072340    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072364    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072435    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072476    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072490    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggwl\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072624    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.072722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072704    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.073058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.072769    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.073058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.073033    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.073168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.073036    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.073246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.073224    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.073436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.073412    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.074711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.074690    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.075214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.075194    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.096415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.096377    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.096564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.096548    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggwl\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.107513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.107486    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:47.122489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.122458    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:47.545543 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:47.545509    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01a0375_83a1_4f1d_b828_d424e6fa2dcf.slice/crio-324d612576f061040a91be0815a96ea6b62210b6d4dfc1fca2449d73679c2780 WatchSource:0}: Error finding container 324d612576f061040a91be0815a96ea6b62210b6d4dfc1fca2449d73679c2780: Status 404 returned error can't find the container with id 324d612576f061040a91be0815a96ea6b62210b6d4dfc1fca2449d73679c2780
Oct 30 07:25:47.546099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.546067    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:25:47.548389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.548351    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.548503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.548422    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.548503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.548456    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.597353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.597308    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:25:47.608244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.608203    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.608564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.608288    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.608564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.608344    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:47.838001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.837889    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 07:25:47.957459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.957424    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" event={"ID":"bb7def7b-3beb-47b2-9d5c-8aa52035eea8","Type":"ContainerStarted","Data":"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"}
Oct 30 07:25:47.957687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.957485    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" event={"ID":"bb7def7b-3beb-47b2-9d5c-8aa52035eea8","Type":"ContainerStarted","Data":"0dce5e60132f89bd2a1cc86f72d54275e68389e608d5b31ea116d58eec084591"}
Oct 30 07:25:47.959648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.959621    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" event={"ID":"a01a0375-83a1-4f1d-b828-d424e6fa2dcf","Type":"ContainerStarted","Data":"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"}
Oct 30 07:25:47.959764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:47.959665    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" event={"ID":"a01a0375-83a1-4f1d-b828-d424e6fa2dcf","Type":"ContainerStarted","Data":"324d612576f061040a91be0815a96ea6b62210b6d4dfc1fca2449d73679c2780"}
Oct 30 07:25:48.004691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.004621    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podStartSLOduration=2.004599802 podStartE2EDuration="2.004599802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:48.003356151 +0000 UTC m=+1708.436120052" watchObservedRunningTime="2026-10-30 07:25:48.004599802 +0000 UTC m=+1708.437363699"
Oct 30 07:25:48.053841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.053789    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podStartSLOduration=2.053771997 podStartE2EDuration="2.053771997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:48.049200058 +0000 UTC m=+1708.481963978" watchObservedRunningTime="2026-10-30 07:25:48.053771997 +0000 UTC m=+1708.486535891"
Oct 30 07:25:48.058751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.058717    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:48.130321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.128149    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:48.132092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.132058    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:48.132327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.132285    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:48.147925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.140631    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:48.148937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.148896    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:48.149164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:48.149135    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:49.111318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:49.110691    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:49.111318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:49.110756    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:49.123071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:49.123033    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:49.123226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:49.123107    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:50.109186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:50.108719    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:50.109186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:50.108785    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:50.123868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:50.123826    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:50.124319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:50.123885    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:50.198685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:50.198648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:51.110816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:51.110767    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:51.111254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:51.111218    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:51.123057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:51.123020    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:51.123243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:51.123137    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:52.118358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:52.117853    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:52.118358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:52.117920    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:52.123604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:52.123571    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:52.123767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:52.123640    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:52.277051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:52.277009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:53.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:53.115728    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:53.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:53.115792    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:53.122969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:53.122929    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:53.123488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:53.122994    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:54.115318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.114346    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:54.115318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.114407    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:54.142321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.130998    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:54.142321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.131078    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:54.464744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.464699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:54.640649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:54.640604    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:55.108265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:55.108227    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 07:25:55.108469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:55.108288    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 07:25:55.123945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:55.123905    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 07:25:55.124120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:55.123976    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 07:25:56.254417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.254378    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:56.254953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.254707    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:56.255954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.255933    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:25:56.356800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.356760    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:56.357859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.357785    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:56.358499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.358476    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:25:56.894027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:56.893985    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:57.094510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.094478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:57.189423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.189265    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:25:57.211065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.211032    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:25:57.230621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.230589    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.303561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.303525    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:25:57.304085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.304066    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:25:57.304283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.304269    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.344035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.343998    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344146    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344179    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344211    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344334    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344380    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ftq\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344448    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344537    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.344617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.344601    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446461    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446513    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.446775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446632    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.446775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446721    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446800    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.446852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446845    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446868    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446873    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.446930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446900    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.446968    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdb98\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447041    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447177    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447210    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447241    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447247    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ftq\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447375    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447448    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447514    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.447556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447542    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.447745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447598    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.448004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.447986    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.449734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.449708    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.449838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.449719    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.491326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.486338    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ftq\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.499986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.499914    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.549413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549360    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549489    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549531    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549649    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549723    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549766    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdb98\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549810    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549831    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549866    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.549961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549927    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.550068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549965    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.550068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.549999    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.550168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.550099    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.550756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.550732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.552122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.552097    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.552261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.552240    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.567213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.567186    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:57.592331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.590460    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdb98\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.595792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.595760    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:57.615908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.615864    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:57.628701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:57.628667    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:58.004305 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:25:58.004252    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b18509_60e3_442f_b08d_d0c42cc6a6d2.slice/crio-3ff675bed729ce562852b5df29732c8c160cb3363bc27859eff673763375f660 WatchSource:0}: Error finding container 3ff675bed729ce562852b5df29732c8c160cb3363bc27859eff673763375f660: Status 404 returned error can't find the container with id 3ff675bed729ce562852b5df29732c8c160cb3363bc27859eff673763375f660
Oct 30 07:25:58.004468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.004334    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:25:58.008623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.008578    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.008730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.008668    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.008730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.008709    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.009814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.009786    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:25:58.012116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.012083    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.012217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.012159    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.012217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.012207    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:25:58.118695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.118560    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:58.147169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.147131    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" event={"ID":"bc3b8adb-b136-4124-81ec-7ef4eba777a8","Type":"ContainerStarted","Data":"631c19825cfe6d258a66ebbc7c67f27f30436db0136b4c5be6499f5d5e3f55ce"}
Oct 30 07:25:58.154283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.154258    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" event={"ID":"d7b18509-60e3-442f-b08d-d0c42cc6a6d2","Type":"ContainerStarted","Data":"3ff675bed729ce562852b5df29732c8c160cb3363bc27859eff673763375f660"}
Oct 30 07:25:58.462334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:58.447215    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:59.166837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.166732    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" event={"ID":"bc3b8adb-b136-4124-81ec-7ef4eba777a8","Type":"ContainerStarted","Data":"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"}
Oct 30 07:25:59.189323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.171365    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" event={"ID":"d7b18509-60e3-442f-b08d-d0c42cc6a6d2","Type":"ContainerStarted","Data":"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"}
Oct 30 07:25:59.239664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.239612    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podStartSLOduration=2.239594212 podStartE2EDuration="2.239594212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:59.226727567 +0000 UTC m=+1719.659491459" watchObservedRunningTime="2026-10-30 07:25:59.239594212 +0000 UTC m=+1719.672358106"
Oct 30 07:25:59.301458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.292117    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podStartSLOduration=2.292090733 podStartE2EDuration="2.292090733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:59.274654595 +0000 UTC m=+1719.707418490" watchObservedRunningTime="2026-10-30 07:25:59.292090733 +0000 UTC m=+1719.724854631"
Oct 30 07:25:59.348218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.348173    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:25:59.617558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.617458    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:25:59.619679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.619646    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:25:59.619902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.619879    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:25:59.632912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.632879    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:25:59.633413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.633382    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:25:59.633668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:25:59.633632    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:00.624340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:00.623456    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:00.624340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:00.623524    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:00.629628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:00.629594    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:00.629807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:00.629652    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:01.624320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.622451    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:01.624320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.622520    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:01.634843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.634803    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:01.635138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.635107    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:01.834886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.834844    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:26:01.841359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.841330    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.865942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.865905    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:26:01.904383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.893638    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.941929    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942014    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942068    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942085    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942124    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgk5\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942144    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942196    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942227    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:01.947361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:01.942339    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.041856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.041816    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044271    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044287    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044331    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgk5\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044349    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044391    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044421    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044492    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044534    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.044886    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.045311    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.045477    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.046335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.046131    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.047933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.047870    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.163261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.134729    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.163261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.163142    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgk5\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.221887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.221794    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.283318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.261321    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.537094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.537004    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:02.629515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.629469    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:02.630026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.629533    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:02.630160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.630133    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:02.630230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:02.630180    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:03.023350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.022657    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:26:03.028190 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:26:03.028149    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b5ac4e_fa12_4488_afc3_f3a92b76f47f.slice/crio-924480d8faa759c01870b9d181231535a757f782677e32dffdc3c093e137e0c9 WatchSource:0}: Error finding container 924480d8faa759c01870b9d181231535a757f782677e32dffdc3c093e137e0c9: Status 404 returned error can't find the container with id 924480d8faa759c01870b9d181231535a757f782677e32dffdc3c093e137e0c9
Oct 30 07:26:03.031835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.031799    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:03.031951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.031869    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:03.031951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.031904    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:03.333883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.333776    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" event={"ID":"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f","Type":"ContainerStarted","Data":"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"}
Oct 30 07:26:03.333883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.333848    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" event={"ID":"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f","Type":"ContainerStarted","Data":"924480d8faa759c01870b9d181231535a757f782677e32dffdc3c093e137e0c9"}
Oct 30 07:26:03.537928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.537885    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:03.639159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.639118    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:03.639747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.639198    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:03.639822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.639806    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:03.639888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.639843    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:03.760035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.747636    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:03.861725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.861680    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:03.916102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.915956    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podStartSLOduration=2.915933626 podStartE2EDuration="2.915933626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:03.395281788 +0000 UTC m=+1723.828045681" watchObservedRunningTime="2026-10-30 07:26:03.915933626 +0000 UTC m=+1724.348697521"
Oct 30 07:26:03.938662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.935368    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 07:26:03.949725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.947918    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 07:26:03.949725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.948134    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:03.975398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.970507    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f5ceca\""
Oct 30 07:26:03.975398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:03.971254    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f5ceca-dockercfg-mtscz\""
Oct 30 07:26:04.054383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.017673    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.085895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.085850    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwrr\" (UniqueName: \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.207409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.195593    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.207409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.195841    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwrr\" (UniqueName: \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.207409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.197653    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.303372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.303240    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:04.406409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.355110    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwrr\" (UniqueName: \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\") pod \"payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.586381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.560990    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:04.614139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.613808    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:04.614826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.614787    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:04.637066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.634794    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:04.637066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.634854    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:04.638549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.638418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:04.656273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.656043    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:04.656273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.656099    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:04.892304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:04.892221    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:05.184759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.184663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:05.590374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.537644    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:05.857501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.835428    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:05.857501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.623898    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:05.857501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.836924    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:05.891331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.874680    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:05.891331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.874743    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:05.936810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:05.936761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:06.062448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.050790    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 07:26:06.544367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.538749    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:06.544367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.538813    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:06.616861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.616821    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 07:26:06.617054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.616871    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 07:26:06.629386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.629347    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 07:26:06.629646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.629399    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 07:26:06.712385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.712339    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" event={"ID":"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2","Type":"ContainerStarted","Data":"bd24311cf41cf9f4c93312d53da0374aafe8b30008d7dd6675b4f7e7decade20"}
Oct 30 07:26:06.712572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.712441    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" event={"ID":"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2","Type":"ContainerStarted","Data":"055dfd703929f3a1576cb8acde0a6ab7a52b03dfe4fe725491e5b09fbaf8849b"}
Oct 30 07:26:06.712572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.712495    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:06.760341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:06.758852    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" podStartSLOduration=3.758830927 podStartE2EDuration="3.758830927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:06.757558177 +0000 UTC m=+1727.190322071" watchObservedRunningTime="2026-10-30 07:26:06.758830927 +0000 UTC m=+1727.191594821"
Oct 30 07:26:07.334515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:07.334474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:07.539274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:07.539230    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:07.539500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:07.539328    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:07.582557 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:26:07.582507    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:26:08.550324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.540222    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:08.550324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.540283    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:08.635331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.634472    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:26:08.635331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.634543    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:08.642315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.642097    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:26:08.642315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.642151    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:08.668342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.667805    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:26:08.677316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.675737    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:26:08.698320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:08.692311    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:26:09.106729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.090119    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:26:09.292128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.201006    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:26:09.292128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.225654    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:26:09.363322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.362852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:09.537395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.535616    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:09.556672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.556633    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:09.585663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.585619    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:09.961315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:09.751495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:10.261379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.260798    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:10.554384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.553777    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:10.554384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.553841    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:10.724760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.724401    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:26:10.724760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.724469    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:10.886160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.849801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:10.886508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:10.886455    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:11.091321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.081230    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:11.216472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.216311    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:11.277409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.242428    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:11.277409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.275384    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:26:11.277409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.275885    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:26:11.385885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.385806    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:11.557660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.557121    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:11.557660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.557181    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:11.569503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:11.569394    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:12.009319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.009218    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:12.037320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.010727    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:12.070814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.070767    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:26:12.203347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.172438    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:12.203347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.172908    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:26:12.203347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.182326    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:26:12.302172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.278405    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:12.620124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.569446    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:12.620124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.569518    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:12.620124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.596939    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:13.011317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.967804    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 07:26:13.012116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:12.983942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:13.012388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.011977    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:13.059615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.059577    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 07:26:13.059974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.059947    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:26:13.060499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.060443    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.080203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.080171    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-137122-openshift-default-dockercfg-22nvl\""
Oct 30 07:26:13.137591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.137545    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.137859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.137840    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.138001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.137983    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.138164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.138149    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.144226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.144189    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.144540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.144521    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.144677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.144663    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.144837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.144820    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.145047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.145025    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbvh\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.192317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.176358    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:13.312367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.311833    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.312367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.312041    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbvh\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.312367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.312239    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.313426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.313120    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.312282    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.313947    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.314035    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.314099    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.314181    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.314825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.314219    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.341179    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.341804    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.342478    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.342740    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.363228    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.388318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.388132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:13.459359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.457446    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:13.459359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.458645    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:13.537318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.533231    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:13.613710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.600631    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 07:26:13.613710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:13.600686    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 07:26:14.098318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.078563    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:14.098318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.079778    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbvh\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\") pod \"e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") " pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:14.098318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.091661    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 07:26:14.155348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.135578    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.156173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.145555    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.341321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.291436    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.341321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.296504    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.341321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.304400    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:14.341321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.309091    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.465920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.465854    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:14.489891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.489849    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 07:26:14.760752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.760621    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 07:26:14.771356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.767105    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:26:14.771356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.767183    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:26:14.818587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.768269    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:14.840835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.840804    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-06e19a-dockercfg-5kdnz\""
Oct 30 07:26:14.947734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.947700    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-06e19a\""
Oct 30 07:26:14.981496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.981394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8kz\" (UniqueName: \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:14.981840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:14.981819    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:15.093615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.093503    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.185549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.099091    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:15.186561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.186535    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8kz\" (UniqueName: \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:15.249196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.108964    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:15.249196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.222407    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.286319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.257248    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.287921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.286566    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.287921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.286940    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:26:15.301346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.295457    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.301346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.296701    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.301346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.296968    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:26:15.316303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.312499    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:26:15.316303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.313689    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:26:15.393428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.363486    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.529176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.492691    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.529176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.528914    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:15.614341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.604412    6738 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:26:15.614341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.604477    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:15.639547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.635908    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:15.664370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.640591    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:15.664370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.651909    6738 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Oct 30 07:26:15.687331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:15.664676    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:26:16.093178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.067361    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8kz\" (UniqueName: \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\") pod \"payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:16.306519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.303518    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.331517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.331481    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 07:26:16.332325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.332271    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:26:16.355782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.340411    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.355782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.340597    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.355782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.352495    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:16.355782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.353165    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d" podUID="2dc94d72-1c5c-414e-b2e5-189467c1edab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded"
Oct 30 07:26:16.363792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.363744    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:16.363952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.363840    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:16.363952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.363890    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:16.366432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.366397    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.366954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.366934    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:26:16.410281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.409436    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.490412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.490375    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 07:26:16.493454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.492905    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:16.712327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.711961    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 07:26:16.792344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.721483    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 07:26:16.792344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.768143    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 07:26:16.792344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.771787    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 07:26:16.826324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.825106    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 07:26:16.835881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.835130    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 07:26:16.839432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.836792    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d"
Oct 30 07:26:16.858618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:16.851413    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 07:26:17.095322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:17.076783    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerStarted","Data":"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"}
Oct 30 07:26:17.095322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:17.076891    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerStarted","Data":"64adff5d7b35302a8f8975fad3600ca32493b4879bb23de6c8131f129ba00eb2"}
Oct 30 07:26:17.235702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:17.235542    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 07:26:17.797682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:17.796341    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 07:26:18.113281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.112957    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:18.136404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.136369    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" event={"ID":"2d4a4a3c-565f-4ffc-ad46-020c713c3f51","Type":"ContainerStarted","Data":"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"}
Oct 30 07:26:18.136667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.136428    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" event={"ID":"2d4a4a3c-565f-4ffc-ad46-020c713c3f51","Type":"ContainerStarted","Data":"abc7a97198fd42d099935020ddd0f7d56054d245763bede7de4dc6fb0977f699"}
Oct 30 07:26:18.136667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.136455    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:18.188169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.188087    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podStartSLOduration=6.188064583 podStartE2EDuration="6.188064583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:18.1799249 +0000 UTC m=+1738.612688794" watchObservedRunningTime="2026-10-30 07:26:18.188064583 +0000 UTC m=+1738.620828477"
Oct 30 07:26:18.227103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.227027    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" podStartSLOduration=5.227005491 podStartE2EDuration="5.227005491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:18.220026978 +0000 UTC m=+1738.652790872" watchObservedRunningTime="2026-10-30 07:26:18.227005491 +0000 UTC m=+1738.659769385"
Oct 30 07:26:18.309428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.309393    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:18.316266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:18.316230    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:19.148392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:19.145780    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:19.148392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:19.147002    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:26:20.160641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.160595    6738 generic.go:356] "Generic (PLEG): container finished" podID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerID="8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a" exitCode=0
Oct 30 07:26:20.161428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.160668    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerDied","Data":"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"}
Oct 30 07:26:20.162358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.162335    6738 scope.go:122] "RemoveContainer" containerID="8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"
Oct 30 07:26:20.168338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.167720    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:20.168338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.167819    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:20.168338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.167862    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:20.309641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.309571    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:20.786965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.786916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:20.908164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:20.908126    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:21.209048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.209007    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerStarted","Data":"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"}
Oct 30 07:26:21.226469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.226430    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:26:21.226674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.226514    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:21.326361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.326245    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:26:21.327674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.327425    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:21.327674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:21.327473    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:22.326251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:22.326167    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:22.326251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:22.326228    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:22.503036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:22.500187    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:22.644986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:22.644945    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:22.847368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:22.847235    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:23.325753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:23.325710    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:23.325945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:23.325765    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:24.265093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.265058    6738 generic.go:356] "Generic (PLEG): container finished" podID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerID="24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26" exitCode=0
Oct 30 07:26:24.265568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.265128    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerDied","Data":"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"}
Oct 30 07:26:24.265682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.265662    6738 scope.go:122] "RemoveContainer" containerID="24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"
Oct 30 07:26:24.267038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267017    6738 generic.go:356] "Generic (PLEG): container finished" podID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerID="1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5" exitCode=0
Oct 30 07:26:24.267137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267080    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerDied","Data":"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"}
Oct 30 07:26:24.284319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267545    6738 scope.go:122] "RemoveContainer" containerID="1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"
Oct 30 07:26:24.284319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267715    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.284319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267801    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.284319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.267851    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.293720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.293668    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.293908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.293784    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.293908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.293836    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.299667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.299640    6738 generic.go:356] "Generic (PLEG): container finished" podID="183fa159-dc30-4e2d-953e-e83ed0986556" containerID="0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69" exitCode=0
Oct 30 07:26:24.299794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.299723    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerDied","Data":"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"}
Oct 30 07:26:24.300321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.300271    6738 scope.go:122] "RemoveContainer" containerID="0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"
Oct 30 07:26:24.307789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.307749    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.307933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.307853    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.307933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.307899    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:24.325770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.325726    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:24.325925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.325787    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:24.632497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.631794    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:26:24.995936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:24.995896    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:25.311073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.310981    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerStarted","Data":"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"}
Oct 30 07:26:25.313609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.313578    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerStarted","Data":"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"}
Oct 30 07:26:25.318619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.318589    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerStarted","Data":"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"}
Oct 30 07:26:25.326667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.326634    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:25.326836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.326696    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:25.344365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.344336    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:26:25.345613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.345576    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:25.345779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.345643    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:25.493319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.492815    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:26:25.495525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.495493    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:25.495785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.495757    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:25.844432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.844396    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:26:25.847328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.844987    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:25.847328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:25.845044    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:26.326991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.326954    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:26.327645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.327612    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:26.346573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.346535    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:26.347599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.347565    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:26.461801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.461763    6738 scope.go:122] "RemoveContainer" containerID="6a0e7c1718f0d76191750548b7439b65ad51f21c14e79cae3221b0a8bd6a01d5"
Oct 30 07:26:26.472865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.472826    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:26.473172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.473145    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:26.547500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.547461    6738 scope.go:122] "RemoveContainer" containerID="b39afb3fcfeea1d046848b077be3355ca05e440e67d2ad61d4cb421a5fb56479"
Oct 30 07:26:26.662593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.662546    6738 scope.go:122] "RemoveContainer" containerID="abd2931c278ef8ab6805d3ab2f90419a821a537fc8e863267b475fbe93ba6593"
Oct 30 07:26:26.664608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.664585    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:26:26.899320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.844525    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:26.899320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.844584    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:26.909357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:26.909202    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:27.365368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.344264    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:27.365368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.344348    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:27.437328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.434582    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 07:26:27.437328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.434684    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 07:26:27.478351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.477620    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:27.478351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.477683    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:27.546672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.536384    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 07:26:27.703623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.703585    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:27.864415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.863556    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:27.864415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:27.863713    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:28.249076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.249034    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:26:28.340635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.340599    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:26:28.341983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.341949    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 07:26:28.346880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.344080    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:28.346880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.346441    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:28.347858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.347834    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 07:26:28.489318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.482695    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:28.489318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.482759    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:28.849317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.847739    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:28.849317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:28.847806    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:29.154319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.152360    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 07:26:29.349378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.345694    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:29.349378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.345757    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:29.356321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.355892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:29.479833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.475230    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:29.479833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.475311    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:29.857314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.857202    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:29.885107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.885044    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:29.886370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:29.886342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:30.211263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.211222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:30.284255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.284219    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 07:26:30.326622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.326590    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.353433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.353400    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-b5998f-openshift-default-dockercfg-h44g7\""
Oct 30 07:26:30.361427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.361394    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:30.386463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.386369    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:30.428428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.389006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:30.428428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.412892    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 07:26:30.468329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.430462    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdhj\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.526321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.518549    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:30.526321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.518615    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:30.598320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.594809    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.598320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.595109    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.598320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.595187    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.598320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.595255    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.598320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.595420    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.635531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.595605    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.635531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.600877    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.635531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.601091    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.719878    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720040    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720087    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720131    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720175    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720238    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720283    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720325    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.721162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.720359    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdhj\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.722707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.722676    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.728369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.724018    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.728763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.728737    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.728916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.728898    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.751316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.734025    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.830396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.830052    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.861354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.861316    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdhj\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.885022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.865846    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.885022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.869351    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:30.885022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.869404    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:30.915319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.907923    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:30.952352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:30.952312    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:31.063938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.063837    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:31.219610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.219575    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:31.348836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.348725    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:31.348836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.348791    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:31.388941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.388911    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:26:31.549418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.548812    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:31.549418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.548875    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:31.822324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.803545    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 07:26:31.847510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.829250    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:31.847510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.829371    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:31.847510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.829419    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:31.889545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.866406    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:31.889545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:31.866470    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:32.133015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.132966    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerStarted","Data":"2772bbf0936dded44f769ec54bea1f7e9d3fac9839a5aee0e6dabae0f79bdc1b"}
Oct 30 07:26:32.356620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.356154    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 07:26:32.356620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.356233    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 07:26:32.478838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.478797    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 07:26:32.479162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.479130    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 07:26:32.825079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.825013    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 07:26:32.845144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.845099    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 07:26:32.845487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.845440    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 07:26:32.855391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.853444    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 07:26:32.855391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.853566    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:32.881335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.873373    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 07:26:32.881335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.873441    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f5ceca-serving-cert\""
Oct 30 07:26:32.896489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.889061    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnjcd\""
Oct 30 07:26:32.897063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.889730    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:26:32.897169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.889813    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:26:32.923347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:32.923311    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:33.043698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.043661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:33.047454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.047287    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.047454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.047372    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c4j\" (UniqueName: \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.047729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.047510    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.047801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.047767    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.150130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.150077    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.306426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.306389    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.307500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.307474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c4j\" (UniqueName: \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.308090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.308067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.308931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.307371    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.309107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.214482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.309234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.184782    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:33.336226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.334968    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.339769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.339731    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerStarted","Data":"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"}
Oct 30 07:26:33.354352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.349420    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c4j\" (UniqueName: \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\") pod \"maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.354352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.353861    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:26:33.355638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.355616    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:26:33.386334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.386215    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podStartSLOduration=3.386165056 podStartE2EDuration="3.386165056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:33.385337566 +0000 UTC m=+1753.818101544" watchObservedRunningTime="2026-10-30 07:26:33.386165056 +0000 UTC m=+1753.818928951"
Oct 30 07:26:33.475080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.474994    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:26:33.476689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.476659    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 07:26:33.501131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.501074    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:33.849439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.849407    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:26:33.850425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.850398    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:26:33.851128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:33.851106    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 07:26:34.011194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.011166    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 07:26:34.013125 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:26:34.013089    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341a8986_9869_4907_8bcf_b66e3dbd71be.slice/crio-6fe74afa6bcfb733b0860b8e22aaa4566ee97304224f8b1247c78e67e02d5874 WatchSource:0}: Error finding container 6fe74afa6bcfb733b0860b8e22aaa4566ee97304224f8b1247c78e67e02d5874: Status 404 returned error can't find the container with id 6fe74afa6bcfb733b0860b8e22aaa4566ee97304224f8b1247c78e67e02d5874
Oct 30 07:26:34.068697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.068631    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:34.081420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.081387    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:34.455282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.452412    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" event={"ID":"341a8986-9869-4907-8bcf-b66e3dbd71be","Type":"ContainerStarted","Data":"6fe74afa6bcfb733b0860b8e22aaa4566ee97304224f8b1247c78e67e02d5874"}
Oct 30 07:26:34.455282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.455126    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:34.465357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:34.465182    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:26:35.324400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:35.324344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:35.460820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:35.460761    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" event={"ID":"341a8986-9869-4907-8bcf-b66e3dbd71be","Type":"ContainerStarted","Data":"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"}
Oct 30 07:26:35.461027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:35.460875    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:35.528286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:35.528235    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" podStartSLOduration=3.048942522 podStartE2EDuration="3.528221301s" totalImagesPullingTime="479.278779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:35.525095518 +0000 UTC m=+1755.957859411" watchObservedRunningTime="2026-10-30 07:26:35.528221301 +0000 UTC m=+1755.960985195"
Oct 30 07:26:35.965261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:35.965229    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:36.090492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:36.090446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:37.521445 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:26:37.521406    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:26:37.629175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.629127    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:37.746706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.746663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:37.832440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.832357    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 07:26:37.841367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.836233    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:37.861225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.861188    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 07:26:37.861951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.861926    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-b5998f-dockercfg-bt4n5\""
Oct 30 07:26:37.862098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.862057    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-b5998f\""
Oct 30 07:26:37.865200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.865169    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:37.865509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.865479    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlxv\" (UniqueName: \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:37.967272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.967231    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlxv\" (UniqueName: \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:37.967472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.967436    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:37.968242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:37.968211    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:38.127665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:38.095751    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlxv\" (UniqueName: \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\") pod \"payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:38.156162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:38.155876    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:38.644104 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:26:38.644069    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ab6451_b0af_4c56_a047_869b89e046c4.slice/crio-590c640a01fc848fb04e643ce8d79c6bc2a3b84537ac21d2b5765a1d03d5490b WatchSource:0}: Error finding container 590c640a01fc848fb04e643ce8d79c6bc2a3b84537ac21d2b5765a1d03d5490b: Status 404 returned error can't find the container with id 590c640a01fc848fb04e643ce8d79c6bc2a3b84537ac21d2b5765a1d03d5490b
Oct 30 07:26:38.645057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:38.645032    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 07:26:39.502787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:39.502750    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" event={"ID":"d7ab6451-b0af-4c56-a047-869b89e046c4","Type":"ContainerStarted","Data":"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"}
Oct 30 07:26:39.503008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:39.502807    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" event={"ID":"d7ab6451-b0af-4c56-a047-869b89e046c4","Type":"ContainerStarted","Data":"590c640a01fc848fb04e643ce8d79c6bc2a3b84537ac21d2b5765a1d03d5490b"}
Oct 30 07:26:39.503008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:39.502849    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:39.545640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:39.545574    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" podStartSLOduration=2.545558847 podStartE2EDuration="2.545558847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:39.542702435 +0000 UTC m=+1759.975466328" watchObservedRunningTime="2026-10-30 07:26:39.545558847 +0000 UTC m=+1759.978322758"
Oct 30 07:26:39.800880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:39.800795    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:41.473960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:41.473930    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 07:26:41.600498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:41.600454    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:41.713774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:41.713736    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:41.832546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:41.832453    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:43.531122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:43.531084    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:45.428746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:45.428704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:47.365480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:47.365445    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:49.196821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:49.196769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:49.316445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:49.316406    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:50.548285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:50.548253    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 07:26:51.379243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.379212    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:51.638139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.638059    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:51.700173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.700124    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"]
Oct 30 07:26:51.819629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.819583    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"]
Oct 30 07:26:51.819820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.819745    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:51.835910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.835885    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:26:51.848371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.848344    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:26:51.934638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.934543    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:51.934638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.934585    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97kq\" (UniqueName: \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:51.934857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:51.934820    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.036407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.036371    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.036407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.036404    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97kq\" (UniqueName: \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.036671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.036636    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.037275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.037257    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.038825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.038807    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.060116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.060088    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97kq\" (UniqueName: \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\") pod \"maas-api-revoke-keys-e2e-autores-04aede-lc8bt\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.137326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.137258    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:52.445974 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:26:52.445904    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee559975_73fa_4c2d_b662_3f5e94c03f81.slice/crio-d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab WatchSource:0}: Error finding container d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab: Status 404 returned error can't find the container with id d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab
Oct 30 07:26:52.446688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.446663    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"]
Oct 30 07:26:52.831726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:52.831636    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt" event={"ID":"ee559975-73fa-4c2d-b662-3f5e94c03f81","Type":"ContainerStarted","Data":"d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab"}
Oct 30 07:26:53.676915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:53.676882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:53.847649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:53.847566    6738 generic.go:356] "Generic (PLEG): container finished" podID="ee559975-73fa-4c2d-b662-3f5e94c03f81" containerID="24d790b5a610bcead1ab2a62d54de99290cce20bf284b45c40cae705016cb0ec" exitCode=0
Oct 30 07:26:53.848066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:53.847649    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt" event={"ID":"ee559975-73fa-4c2d-b662-3f5e94c03f81","Type":"ContainerDied","Data":"24d790b5a610bcead1ab2a62d54de99290cce20bf284b45c40cae705016cb0ec"}
Oct 30 07:26:54.420782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.420743    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:26:54.424962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.424930    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.433866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.433599    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:26:54.438089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.438061    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.451919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.451888    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:26:54.470477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.470438    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:26:54.580683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.580649    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.580886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.580698    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.580886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.580815    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.580886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.580874    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.580896    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581005    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfgg\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581047    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581085    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581137    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581165    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581268    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581406    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581432    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581465    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr9k\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581590    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581619    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.581690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581655    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.581747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.581726    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.682964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.682880    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.682964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.682914    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.682964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.682946    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfgg\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.682975    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683003    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683019    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683040    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683142    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683168    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683210    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr9k\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683332    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683356    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683455    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683483    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683492    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683560    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683604    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683642    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.683795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683669    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683930    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.683953    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.684108    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.684340    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.684390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.684375    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.684758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.684657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.685980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.685951    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.686234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.686214    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.686340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.686239    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.686419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.686403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.706982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.706952    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.707131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.706952    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfgg\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:54.707131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.707003    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.707131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.707119    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr9k\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.751508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.751465    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:54.788767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:54.788738    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:55.081776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.081743    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:55.082544 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:26:55.082518    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882b7b57_7108_4428_9ab0_5a5397226c00.slice/crio-08fc4fce07554e23b4c8da28e075f01007ec7ad9c6acd2ab562456187364f41f WatchSource:0}: Error finding container 08fc4fce07554e23b4c8da28e075f01007ec7ad9c6acd2ab562456187364f41f: Status 404 returned error can't find the container with id 08fc4fce07554e23b4c8da28e075f01007ec7ad9c6acd2ab562456187364f41f
Oct 30 07:26:55.083134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.083113    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:26:55.085613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.085577    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.085692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.085647    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.085692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.085677    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.121863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.121832    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:26:55.130740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.130702    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.130871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.130815    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.130871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.130862    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:26:55.193723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.193689    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\") pod \"ee559975-73fa-4c2d-b662-3f5e94c03f81\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") "
Oct 30 07:26:55.193890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.193756    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\" (UniqueName: \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\") pod \"ee559975-73fa-4c2d-b662-3f5e94c03f81\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") "
Oct 30 07:26:55.194049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.194032    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\") pod \"ee559975-73fa-4c2d-b662-3f5e94c03f81\" (UID: \"ee559975-73fa-4c2d-b662-3f5e94c03f81\") "
Oct 30 07:26:55.194401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.194376    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca" pod "ee559975-73fa-4c2d-b662-3f5e94c03f81" (UID: "ee559975-73fa-4c2d-b662-3f5e94c03f81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:26:55.194782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.194761    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:26:55.196036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.196004    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq" pod "ee559975-73fa-4c2d-b662-3f5e94c03f81" (UID: "ee559975-73fa-4c2d-b662-3f5e94c03f81"). InnerVolumeSpecName "kube-api-access-t97kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:26:55.196499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.196476    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca" pod "ee559975-73fa-4c2d-b662-3f5e94c03f81" (UID: "ee559975-73fa-4c2d-b662-3f5e94c03f81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:26:55.296738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.296613    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee559975-73fa-4c2d-b662-3f5e94c03f81-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:26:55.296738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.296649    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97kq\" (UniqueName: \"kubernetes.io/projected/ee559975-73fa-4c2d-b662-3f5e94c03f81-kube-api-access-t97kq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:26:55.536504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.536464    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:55.929937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.929902    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"
Oct 30 07:26:55.930404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.930376    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt" event={"ID":"ee559975-73fa-4c2d-b662-3f5e94c03f81","Type":"ContainerDied","Data":"d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab"}
Oct 30 07:26:55.930501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.930431    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31d1dbb7ad1a14a6df0d97fd54d4660b8e41c99ae26b13a4ef35ed6e9aed8ab"
Oct 30 07:26:55.932969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.932943    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" event={"ID":"882b7b57-7108-4428-9ab0-5a5397226c00","Type":"ContainerStarted","Data":"ab29800fdfb76a0870e0ddfa63bf2adcedc90bef01b0fa3deb3ec19410bf4ae4"}
Oct 30 07:26:55.933082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.933001    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" event={"ID":"882b7b57-7108-4428-9ab0-5a5397226c00","Type":"ContainerStarted","Data":"08fc4fce07554e23b4c8da28e075f01007ec7ad9c6acd2ab562456187364f41f"}
Oct 30 07:26:55.934908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.934885    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" event={"ID":"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75","Type":"ContainerStarted","Data":"5f62509eec1beea140e1f1c6770a2f52f3a1e7e775dc9199903dd22dba35f1c2"}
Oct 30 07:26:55.935014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:55.934921    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" event={"ID":"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75","Type":"ContainerStarted","Data":"f063e9eb6dfab6db42cf6c8020da57e8d245af622a85be710768387efa376a15"}
Oct 30 07:26:56.064181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.064066    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podStartSLOduration=2.064045953 podStartE2EDuration="2.064045953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:56.01726293 +0000 UTC m=+1776.450026827" watchObservedRunningTime="2026-10-30 07:26:56.064045953 +0000 UTC m=+1776.496809847"
Oct 30 07:26:56.065123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.065086    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podStartSLOduration=2.065077234 podStartE2EDuration="2.065077234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:56.061882816 +0000 UTC m=+1776.494646710" watchObservedRunningTime="2026-10-30 07:26:56.065077234 +0000 UTC m=+1776.497841196"
Oct 30 07:26:56.751631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.751597    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:26:56.753423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.753396    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 07:26:56.753547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.753449    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 07:26:56.797329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.794361    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:26:56.809334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.798646    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 07:26:56.809334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:56.798710    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 07:26:57.643898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:57.643856    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:26:57.761332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:57.759470    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 07:26:57.761332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:57.759550    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 07:26:57.802354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:57.792874    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 07:26:57.802354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:57.792935    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 07:26:58.752566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:58.752502    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 07:26:58.752794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:58.752589    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 07:26:58.791316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:58.790685    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 07:26:58.791316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:58.790758    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 07:26:59.751951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:59.751921    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 07:26:59.752210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:59.752187    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 07:26:59.791724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:59.791684    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 07:26:59.792342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:59.792280    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 07:26:59.964857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:26:59.964822    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:00.752650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:00.752612    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 30 07:27:00.752856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:00.752674    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 30 07:27:00.796983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:00.796944    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 07:27:00.797738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:00.797700    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 07:27:01.757698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.757666    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:27:01.757979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.757955    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:27:01.758781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.758762    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:27:01.793979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.793951    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:27:01.794389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.794363    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:27:01.795220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.795200    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:27:01.829731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.829684    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:01.954433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:01.954337    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:02.111600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:02.111416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:02.455393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:02.455344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:02.593968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:02.586068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:04.320416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:04.320375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:04.470620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:04.470578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:04.871677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:04.871641    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:06.832615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:06.832578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:07.535162 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:27:07.535122    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:27:08.292638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:08.292587    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:08.793941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:08.793899    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:10.793946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:10.793904    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:12.746452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:12.746354    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:14.928533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:14.928495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:15.849438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:15.849401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:16.507186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:16.507145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:16.674598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:16.674564    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:17.392107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.391923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:17.449602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.449552    6738 generic.go:356] "Generic (PLEG): container finished" podID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerID="dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6" exitCode=0
Oct 30 07:27:17.450060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.449839    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerDied","Data":"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"}
Oct 30 07:27:17.464657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.464627    6738 scope.go:122] "RemoveContainer" containerID="dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"
Oct 30 07:27:17.472707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.472630    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:17.473702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.473656    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:17.474014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.473955    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:17.545646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.545540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:17.614895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.614848    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"]
Oct 30 07:27:17.621815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.621764    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04aede-lc8bt"]
Oct 30 07:27:17.656851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.656748    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:17.800453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:17.800418    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee559975-73fa-4c2d-b662-3f5e94c03f81" path="/var/lib/kubelet/pods/ee559975-73fa-4c2d-b662-3f5e94c03f81/volumes"
Oct 30 07:27:18.167610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:18.167567    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:18.461414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:18.461321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerStarted","Data":"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"}
Oct 30 07:27:19.146031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:19.145992    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:27:19.308935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:19.308901    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:27:19.310285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:19.310259    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:19.310439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:19.310322    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:19.896213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:19.896126    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:20.308801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:20.308708    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:20.309422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:20.309386    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:21.308709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:21.308669    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:21.308924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:21.308744    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:22.101813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.101770    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:22.102872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.102844    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ee559975-73fa-4c2d-b662-3f5e94c03f81" containerName="revoke-keys"
Oct 30 07:27:22.102872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.102867    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee559975-73fa-4c2d-b662-3f5e94c03f81" containerName="revoke-keys"
Oct 30 07:27:22.103058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.102984    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee559975-73fa-4c2d-b662-3f5e94c03f81" containerName="revoke-keys"
Oct 30 07:27:22.106492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.106473    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.129872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.129836    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:22.134603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.134572    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-04aede-openshift-default-dockercfg-qhx9g\""
Oct 30 07:27:22.141967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.141925    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzjr\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142030    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142101    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142195    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142249    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142273    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142364    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142425    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.142536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.142464    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.151485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.151448    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:22.244681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.244628    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.244871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.244804    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.244920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.244868    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.244920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.244897    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.244943    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.245014    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.245060    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.245142    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzjr\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.245281    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.245927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.245897    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.246254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.246151    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.265331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.246813    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.265331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.247076    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.265331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.250165    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.316319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.295397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.316319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.311586    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.332137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.330894    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.332137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.331015    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:22.332137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.331057    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:22.332137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.332118    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzjr\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\") pod \"e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") " pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.424316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.422593    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:22.900681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.900646    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:22.901159 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:27:22.901130    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9507970_4c15_45f1_a322_332eb9f1728b.slice/crio-06233c24963057e940a9fc155ffe44df7e680cc6c13ff390697d1d3f48455695 WatchSource:0}: Error finding container 06233c24963057e940a9fc155ffe44df7e680cc6c13ff390697d1d3f48455695: Status 404 returned error can't find the container with id 06233c24963057e940a9fc155ffe44df7e680cc6c13ff390697d1d3f48455695
Oct 30 07:27:22.904476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.904439    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:22.904561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.904513    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:22.904561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:22.904545    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:23.331321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:23.316640    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:23.331321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:23.316699    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:23.641974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:23.640165    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" event={"ID":"e9507970-4c15-45f1-a322-332eb9f1728b","Type":"ContainerStarted","Data":"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"}
Oct 30 07:27:23.641974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:23.640234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" event={"ID":"e9507970-4c15-45f1-a322-332eb9f1728b","Type":"ContainerStarted","Data":"06233c24963057e940a9fc155ffe44df7e680cc6c13ff390697d1d3f48455695"}
Oct 30 07:27:23.765316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:23.764397    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" podStartSLOduration=1.764375743 podStartE2EDuration="1.764375743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:23.703393465 +0000 UTC m=+1804.136157359" watchObservedRunningTime="2026-10-30 07:27:23.764375743 +0000 UTC m=+1804.197139637"
Oct 30 07:27:24.316315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:24.315738    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 07:27:24.316315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:24.315808    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 07:27:24.447383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:24.427477    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:24.447383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:24.444436    6738 patch_prober.go:29] interesting pod/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 07:27:24.447383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:24.444503    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 07:27:25.023353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.023238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:25.187321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.174656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:25.339770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.339649    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:25.430764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.430724    6738 patch_prober.go:29] interesting pod/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 07:27:25.431658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.431616    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 07:27:25.546904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.546868    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:25.615372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.613866    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:25.942319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.935135    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:27:26.016317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:25.993483    6738 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" secret="" err="secret \"e2e-autores-04aede-openshift-default-dockercfg-qhx9g\" not found"
Oct 30 07:27:26.078319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:26.020541    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 07:27:26.246976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:26.246871    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:26.424041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:26.424002    6738 patch_prober.go:29] interesting pod/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 07:27:26.424345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:26.424080    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 07:27:26.763358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:26.763318    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:27.051379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.049761    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy" containerID="cri-o://7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235" gracePeriod=30
Oct 30 07:27:27.263555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.263502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:27.386125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.385947    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:27.649831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.649745    6738 scope.go:122] "RemoveContainer" containerID="1f36aca6b883caf80f979683c99a5d879d4f7a89c8bb1e2445ba0ecb440c12fc"
Oct 30 07:27:27.662043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.662021    6738 scope.go:122] "RemoveContainer" containerID="3d50c3ee5f52d665fddab44037f722d15c88fb5e3e19c55c3891a655a865c256"
Oct 30 07:27:27.739923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:27.739888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:29.821376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:29.819975    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:30.037040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:30.036999    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"]
Oct 30 07:27:30.045452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:30.045417    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4231-f844s"]
Oct 30 07:27:30.840136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:30.840096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:30.962623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:30.962581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:31.116470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:31.116385    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:31.694412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:31.694374    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967f22d4-146a-49a2-b045-d5fa3b42fb26" path="/var/lib/kubelet/pods/967f22d4-146a-49a2-b045-d5fa3b42fb26/volumes"
Oct 30 07:27:32.018117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:32.018008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:32.606765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:32.606726    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:32.780627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:32.780591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:33.128372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.128191    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:27:33.240118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.240084    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.253682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.252481    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:27:33.281164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.281134    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f84d9e\""
Oct 30 07:27:33.281164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.281136    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f84d9e-dockercfg-sfncn\""
Oct 30 07:27:33.359375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.359340    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2sb\" (UniqueName: \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.359618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.359596    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.461396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.461039    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.461396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.461328    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2sb\" (UniqueName: \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.462026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.462004    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.484921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.484888    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2sb\" (UniqueName: \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\") pod \"payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") " pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.573124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.573088    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:33.884802 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:27:33.884761    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbbcb7b_df22_4aca_a8e5_eb472c6ed30d.slice/crio-da607d846974fe396f804c874cbaf64d9c37b501bee4a80da16b5728932f8c41 WatchSource:0}: Error finding container da607d846974fe396f804c874cbaf64d9c37b501bee4a80da16b5728932f8c41: Status 404 returned error can't find the container with id da607d846974fe396f804c874cbaf64d9c37b501bee4a80da16b5728932f8c41
Oct 30 07:27:33.888361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:33.888255    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:27:34.231693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.231608    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:34.361487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.360801    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" event={"ID":"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d","Type":"ContainerStarted","Data":"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"}
Oct 30 07:27:34.361487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.360874    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" event={"ID":"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d","Type":"ContainerStarted","Data":"da607d846974fe396f804c874cbaf64d9c37b501bee4a80da16b5728932f8c41"}
Oct 30 07:27:34.370407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.366623    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:34.421073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.421020    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" podStartSLOduration=2.421003674 podStartE2EDuration="2.421003674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:34.406622942 +0000 UTC m=+1814.839386836" watchObservedRunningTime="2026-10-30 07:27:34.421003674 +0000 UTC m=+1814.853767567"
Oct 30 07:27:34.810574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.810537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:34.934497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.934449    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:34.954799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.954658    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": read tcp 10.133.0.2:53580->10.133.1.20:15021: read: connection reset by peer" start-of-body=
Oct 30 07:27:34.954799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:34.954713    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": read tcp 10.133.0.2:53580->10.133.1.20:15021: read: connection reset by peer"
Oct 30 07:27:35.371886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.371854    6738 generic.go:356] "Generic (PLEG): container finished" podID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerID="d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c" exitCode=0
Oct 30 07:27:35.372424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.371934    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerDied","Data":"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"}
Oct 30 07:27:35.372570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.372551    6738 scope.go:122] "RemoveContainer" containerID="d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"
Oct 30 07:27:35.374203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.374175    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:35.374282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.374242    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:35.374282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:35.374275    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:27:36.076848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.076810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:36.274572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.274538    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:36.377183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377146    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377303    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377383    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377421    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377501    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377543    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377649    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.377748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377696    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.378197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377827    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\") pod \"e9507970-4c15-45f1-a322-332eb9f1728b\" (UID: \"e9507970-4c15-45f1-a322-332eb9f1728b\") "
Oct 30 07:27:36.378197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.377989    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:36.378197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.378189    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:36.379171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.378734    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.379171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.378763    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.379171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.378802    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:27:36.379171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.378991    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:36.379171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.379128    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:36.380332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.380270    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:27:36.380870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.380844    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:36.381412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.381385    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "kube-api-access-6dzjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:27:36.381733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.381713    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token" pod "e9507970-4c15-45f1-a322-332eb9f1728b" (UID: "e9507970-4c15-45f1-a322-332eb9f1728b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:27:36.387265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.387242    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerStarted","Data":"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"}
Oct 30 07:27:36.389278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.389257    6738 generic.go:356] "Generic (PLEG): container finished" podID="e9507970-4c15-45f1-a322-332eb9f1728b" containerID="7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235" exitCode=0
Oct 30 07:27:36.389540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.389328    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"
Oct 30 07:27:36.389540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.389396    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" event={"ID":"e9507970-4c15-45f1-a322-332eb9f1728b","Type":"ContainerDied","Data":"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"}
Oct 30 07:27:36.389540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.389456    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs" event={"ID":"e9507970-4c15-45f1-a322-332eb9f1728b","Type":"ContainerDied","Data":"06233c24963057e940a9fc155ffe44df7e680cc6c13ff390697d1d3f48455695"}
Oct 30 07:27:36.389540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.389486    6738 scope.go:122] "RemoveContainer" containerID="7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"
Oct 30 07:27:36.416021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.415998    6738 scope.go:122] "RemoveContainer" containerID="7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"
Oct 30 07:27:36.416398 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:27:36.416372    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235\": container with ID starting with 7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235 not found: ID does not exist" containerID="7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"
Oct 30 07:27:36.416516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.416409    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235"} err="failed to get container status \"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235\": rpc error: code = NotFound desc = could not find container \"7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235\": container with ID starting with 7514274fd643e80a6a0a6560def1c017dbc78f99a19810ca542bf12b33a68235 not found: ID does not exist"
Oct 30 07:27:36.498710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498616    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498662    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9507970-4c15-45f1-a322-332eb9f1728b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498680    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498696    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzjr\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-kube-api-access-6dzjr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498715    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9507970-4c15-45f1-a322-332eb9f1728b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498735    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9507970-4c15-45f1-a322-332eb9f1728b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.498979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.498749    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9507970-4c15-45f1-a322-332eb9f1728b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:27:36.518973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.518931    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:36.523334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:36.523288    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-04aede-openshift-default-6d56d6b7fc-8g9hs"]
Oct 30 07:27:37.076320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:37.071723    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:27:37.076320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:37.071859    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:37.076320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:37.071904    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:37.521507 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:27:37.521474    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:27:37.700674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:37.700642    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" path="/var/lib/kubelet/pods/e9507970-4c15-45f1-a322-332eb9f1728b/volumes"
Oct 30 07:27:38.081348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:38.075705    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:38.081348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:38.075769    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:38.387898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:38.387863    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:39.024847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:39.024803    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:39.079328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:39.074683    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:39.079328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:39.074748    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:40.082855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:40.082806    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:40.083496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:40.083461    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:40.708429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:40.708392    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:41.086380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:41.086242    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:41.086380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:41.086335    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:42.079426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:42.079384    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 07:27:42.079736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:42.079698    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 07:27:42.550309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:42.550260    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:43.000507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:43.000471    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:43.130704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:43.130670    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:27:43.132881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:43.132838    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:27:43.140465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:43.140429    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 07:27:44.230258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:44.230214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:45.036997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:45.036953    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:46.393350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:46.393279    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:27:46.932812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:46.932777    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:48.727832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:48.727796    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:50.535676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:50.535639    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:52.354226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:52.354187    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:54.241981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:54.241943    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:55.834980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:55.834927    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:56.150901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:56.150864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:58.048515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:58.048477    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:58.179451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:58.175578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:58.290783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:58.290747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:59.867106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.867070    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:27:59.867997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.867977    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy"
Oct 30 07:27:59.868051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.868000    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy"
Oct 30 07:27:59.868123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.868113    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9507970-4c15-45f1-a322-332eb9f1728b" containerName="istio-proxy"
Oct 30 07:27:59.870948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.870928    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.887190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.887160    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:27:59.895628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.895598    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"default-dockercfg-p7lgh\""
Oct 30 07:27:59.895852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.895676    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"auth-test-model-f5ceca-kserve-self-signed-certs\""
Oct 30 07:27:59.909235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.909195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:27:59.970993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.970961    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"openshift-service-ca.crt\""
Oct 30 07:27:59.970993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.970966    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"kube-root-ca.crt\""
Oct 30 07:27:59.994945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.994906    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.995149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.995113    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.995200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.995147    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.995359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.995275    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.995473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.995461    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:27:59.995513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:27:59.995486    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kqg\" (UniqueName: \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.079959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.079916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:00.096684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096655    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.096684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096687    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.096918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096778    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.096918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096884    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.096918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096911    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kqg\" (UniqueName: \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.097067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.096951    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.097189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.097160    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.097260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.097168    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.097442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.097423    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.099268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.099241    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.099436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.099419    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.158439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.158343    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kqg\" (UniqueName: \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\") pod \"auth-test-model-f5ceca-kserve-56d89f569-9fpj4\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.191800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.191686    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:00.650910 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:28:00.650879    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48fd1a4d_fdb7_426d_8381_14ef5874473c.slice/crio-4e289adc1d0d1a99892c79b96ea9c35c96ca1e9c0ca4970f6dae38448b2a1ed4 WatchSource:0}: Error finding container 4e289adc1d0d1a99892c79b96ea9c35c96ca1e9c0ca4970f6dae38448b2a1ed4: Status 404 returned error can't find the container with id 4e289adc1d0d1a99892c79b96ea9c35c96ca1e9c0ca4970f6dae38448b2a1ed4
Oct 30 07:28:00.651713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.651688    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:28:00.767195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.767150    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" event={"ID":"48fd1a4d-fdb7-426d-8381-14ef5874473c","Type":"ContainerStarted","Data":"d56df7d2e88f6a1aa01884889d47257473e3ed13eac1ba108ce2bf65e63d9d91"}
Oct 30 07:28:00.767426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.767204    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" event={"ID":"48fd1a4d-fdb7-426d-8381-14ef5874473c","Type":"ContainerStarted","Data":"4e289adc1d0d1a99892c79b96ea9c35c96ca1e9c0ca4970f6dae38448b2a1ed4"}
Oct 30 07:28:00.810204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:00.810149    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" podStartSLOduration=1.810134577 podStartE2EDuration="1.810134577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:00.80851978 +0000 UTC m=+1841.241283675" watchObservedRunningTime="2026-10-30 07:28:00.810134577 +0000 UTC m=+1841.242898469"
Oct 30 07:28:01.192059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:01.192026    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:02.027659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:02.027623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:03.783598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:03.783567    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:05.904333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:05.904280    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:06.294387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:06.294286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:06.404903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:06.404866    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:07.519441 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:28:07.519404    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:28:07.744833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:07.744802    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:08.451996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.451962    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:28:08.456178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.456155    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.471491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.471391    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bd8589\"/\"kube-root-ca.crt\""
Oct 30 07:28:08.475015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.474981    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:28:08.481776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.481741    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bd8589\"/\"auth-test-model-bd8589-kserve-self-signed-certs\""
Oct 30 07:28:08.482072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.481841    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bd8589\"/\"openshift-service-ca.crt\""
Oct 30 07:28:08.486434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.486404    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bd8589\"/\"default-dockercfg-fd269\""
Oct 30 07:28:08.488561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.488527    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.488724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.488609    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.488834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.488671    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.488834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.488797    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.488980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.488953    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2lh\" (UniqueName: \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.489228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.489189    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.590784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.590748    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.590870    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.590908    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.590942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.590978    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.591030    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2lh\" (UniqueName: \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.591235    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.591258    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.591601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.591475    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.593229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.593202    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.593650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.593631    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.638197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.638158    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2lh\" (UniqueName: \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\") pod \"auth-test-model-bd8589-kserve-854bf69c65-g4xk5\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") " pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:08.696044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.696006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:08.777130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:08.777042    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:09.330729 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:28:09.330697    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47eaf0fa_b417_4d9c_a08e_cbc2a188c418.slice/crio-f7f2382c695f871f91227abc3ad37ba8f0002d64aef40e7348502ee5cb2be6a5 WatchSource:0}: Error finding container f7f2382c695f871f91227abc3ad37ba8f0002d64aef40e7348502ee5cb2be6a5: Status 404 returned error can't find the container with id f7f2382c695f871f91227abc3ad37ba8f0002d64aef40e7348502ee5cb2be6a5
Oct 30 07:28:09.331485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:09.331455    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:28:09.745899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:09.745860    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:10.008825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.008728    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" event={"ID":"47eaf0fa-b417-4d9c-a08e-cbc2a188c418","Type":"ContainerStarted","Data":"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"}
Oct 30 07:28:10.008825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.008780    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" event={"ID":"47eaf0fa-b417-4d9c-a08e-cbc2a188c418","Type":"ContainerStarted","Data":"f7f2382c695f871f91227abc3ad37ba8f0002d64aef40e7348502ee5cb2be6a5"}
Oct 30 07:28:10.043676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.043643    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"]
Oct 30 07:28:10.048260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.048226    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0190638-v2gjn"]
Oct 30 07:28:10.067649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.067595    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" podStartSLOduration=2.067580928 podStartE2EDuration="2.067580928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:10.065774129 +0000 UTC m=+1850.498538022" watchObservedRunningTime="2026-10-30 07:28:10.067580928 +0000 UTC m=+1850.500344883"
Oct 30 07:28:10.199467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.199428    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:10.246323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.242808    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:10.777392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:10.777359    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:11.028800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:11.028724    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:28:11.633011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:11.632972    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:11.756580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:11.755580    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d7896a-e8eb-4303-bb51-75295a779c7f" path="/var/lib/kubelet/pods/06d7896a-e8eb-4303-bb51-75295a779c7f/volumes"
Oct 30 07:28:13.534741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:13.534699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:14.885856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:14.885822    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:15.618562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:15.618523    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:17.433163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:17.433122    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:18.777645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:18.777598    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:18.791151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:18.791122    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:19.200518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:19.200486    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:28:19.230642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:19.230607    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:21.224394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:21.224353    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:23.452726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:23.452678    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:24.457057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:24.457015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:25.247202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:25.247164    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:27.042339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:27.042284    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:27.972213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:27.972178    6738 scope.go:122] "RemoveContainer" containerID="2ecbaca7c2d69fd04ff9f83d6051aaefcf37ec9bfdc107785a2a23d0412afcfc"
Oct 30 07:28:27.984192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:27.984166    6738 scope.go:122] "RemoveContainer" containerID="a1626e082256d66e97e5312618af2a0df510d049cefdcf868c778c68a6de50b8"
Oct 30 07:28:27.995044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:27.995023    6738 scope.go:122] "RemoveContainer" containerID="59b0d9cc6ee3f399c2e67f4107ee27d13f50acdad0169a63e884286702e93aed"
Oct 30 07:28:28.006072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:28.006051    6738 scope.go:122] "RemoveContainer" containerID="1adccc871629592f2efd4b7ec492445ed8b0a56ba4c505c01728ef47f38c781e"
Oct 30 07:28:29.187451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:29.187410    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:30.544654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:30.544606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:31.255915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:31.255878    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:31.381691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:31.381650    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:32.196186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:32.196153    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:33.166951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:33.166912    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:35.169744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:35.169708    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:37.133740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:37.133705    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:37.519647 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:28:37.519563    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:28:39.041669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:39.041630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:41.061634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:41.061600    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"]
Oct 30 07:28:41.093317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:41.092618    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:41.121671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:41.121626    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4640d98d-42rvv"]
Oct 30 07:28:41.692215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:41.692181    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fd9a3f-232e-482b-907d-6fd3ed9fceaa" path="/var/lib/kubelet/pods/23fd9a3f-232e-482b-907d-6fd3ed9fceaa/volumes"
Oct 30 07:28:42.977850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:42.977811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:44.847275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:44.847236    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:46.694979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:46.694942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:47.421278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:47.421232    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:48.761823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:48.761789    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:50.770318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:50.764622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:52.648746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:52.648697    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:54.030034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:54.029997    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"]
Oct 30 07:28:54.042280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:54.042248    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9f8fc8-4xb5c"]
Oct 30 07:28:54.524364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:54.524325    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:55.692005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:55.691968    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24131829-0c28-4fdc-bd4e-e12db24c6e85" path="/var/lib/kubelet/pods/24131829-0c28-4fdc-bd4e-e12db24c6e85/volumes"
Oct 30 07:28:56.410731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:56.410689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:28:58.185849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:28:58.185812    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:00.045038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:00.044949    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:02.196010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:02.195970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:03.651857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:03.651815    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:04.439353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:04.439273    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:05.656959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.656921    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:29:05.663208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.663181    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.698344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.691177    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-06e19a-openshift-default-dockercfg-7ptkv\""
Oct 30 07:29:05.698344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.696626    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:29:05.782922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.782888    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.782979    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783006    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783185    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783221    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783379    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783558    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783595    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.783730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.783713    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf97q\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885228    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885327    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885418    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885479    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885506    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885574    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885708    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885743    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885830    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885848    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf97q\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.885906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.885836    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.886197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.886176    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.886370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.886350    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.886510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.886493    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.888373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.888352    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.888728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.888698    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.909493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.909421    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.909649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.909625    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf97q\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\") pod \"e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") " pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:05.979993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:05.979956    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:06.311199 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:29:06.311160    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e290c_60b9_4347_a3ef_ca6242c0cf75.slice/crio-ec0242f2dbd524a67c79ad1b45adc06ae536d93db244dbc8a03da2e664f0c260 WatchSource:0}: Error finding container ec0242f2dbd524a67c79ad1b45adc06ae536d93db244dbc8a03da2e664f0c260: Status 404 returned error can't find the container with id ec0242f2dbd524a67c79ad1b45adc06ae536d93db244dbc8a03da2e664f0c260
Oct 30 07:29:06.312009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.311981    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:29:06.314193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.314157    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:29:06.314345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.314234    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:29:06.314345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.314282    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:29:06.415735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.415702    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:06.827357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.827313    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" event={"ID":"204e290c-60b9-4347-a3ef-ca6242c0cf75","Type":"ContainerStarted","Data":"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"}
Oct 30 07:29:06.827921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.827375    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" event={"ID":"204e290c-60b9-4347-a3ef-ca6242c0cf75","Type":"ContainerStarted","Data":"ec0242f2dbd524a67c79ad1b45adc06ae536d93db244dbc8a03da2e664f0c260"}
Oct 30 07:29:06.980258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:06.980230    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:07.032015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.031959    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podStartSLOduration=2.03194316 podStartE2EDuration="2.03194316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:06.878554936 +0000 UTC m=+1907.311318833" watchObservedRunningTime="2026-10-30 07:29:07.03194316 +0000 UTC m=+1907.464707053"
Oct 30 07:29:07.032488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.032466    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"]
Oct 30 07:29:07.036821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.036798    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4327f18-7xd55"]
Oct 30 07:29:07.522721 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:29:07.522679    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:29:07.700732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.700694    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7366184e-cb74-4f47-9d7a-afc8102053ea" path="/var/lib/kubelet/pods/7366184e-cb74-4f47-9d7a-afc8102053ea/volumes"
Oct 30 07:29:07.995551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.995512    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:07.996173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:07.996143    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:08.318690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:08.318598    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:08.992317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:08.991405    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:08.992317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:08.991486    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:09.930559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:09.930516    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:09.982747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:09.982705    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:09.985920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:09.985379    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:10.422045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.422002    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"]
Oct 30 07:29:10.427441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.427418    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.444793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.444760    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"]
Oct 30 07:29:10.449679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.449642    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:29:10.449890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.449782    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:29:10.502555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.502513    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.502776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.502637    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.502776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.502722    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9px\" (UniqueName: \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.605049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.605013    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.605247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.605070    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.605247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.605112    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9px\" (UniqueName: \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.605679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.605656    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.607907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.607423    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.628992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.628953    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9px\" (UniqueName: \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\") pod \"maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.733100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.733008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:10.746731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.746687    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:10.990377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.983730    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:10.990377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:10.983796    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:11.265376 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:29:11.265343    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583372e5_9c7c_4e43_874f_91e94cf652ad.slice/crio-11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26 WatchSource:0}: Error finding container 11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26: Status 404 returned error can't find the container with id 11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26
Oct 30 07:29:11.266464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.266440    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"]
Oct 30 07:29:11.951359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.951282    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc" event={"ID":"583372e5-9c7c-4e43-874f-91e94cf652ad","Type":"ContainerStarted","Data":"270aa5edf1d8b4890363d9a607d96c419096ee7110378d2d0c2a9ff050c59367"}
Oct 30 07:29:11.951568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.951388    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc" event={"ID":"583372e5-9c7c-4e43-874f-91e94cf652ad","Type":"ContainerStarted","Data":"11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26"}
Oct 30 07:29:11.994098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.993936    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:11.994098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.994000    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:11.994743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:11.994391    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc" podStartSLOduration=1.994379093 podStartE2EDuration="1.994379093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:11.994195292 +0000 UTC m=+1912.426959185" watchObservedRunningTime="2026-10-30 07:29:11.994379093 +0000 UTC m=+1912.427142986"
Oct 30 07:29:12.830781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:12.830692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:12.980672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:12.980635    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 07:29:12.981494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:12.981444    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 07:29:12.983682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:12.983653    6738 generic.go:356] "Generic (PLEG): container finished" podID="583372e5-9c7c-4e43-874f-91e94cf652ad" containerID="270aa5edf1d8b4890363d9a607d96c419096ee7110378d2d0c2a9ff050c59367" exitCode=0
Oct 30 07:29:12.984160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:12.983799    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc" event={"ID":"583372e5-9c7c-4e43-874f-91e94cf652ad","Type":"ContainerDied","Data":"270aa5edf1d8b4890363d9a607d96c419096ee7110378d2d0c2a9ff050c59367"}
Oct 30 07:29:13.985978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:13.985948    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:13.986496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:13.986482    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:13.987358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:13.987334    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:29:14.212094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.212063    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:14.406992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.406948    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\") pod \"583372e5-9c7c-4e43-874f-91e94cf652ad\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") "
Oct 30 07:29:14.407209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.407159    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\" (UniqueName: \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\") pod \"583372e5-9c7c-4e43-874f-91e94cf652ad\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") "
Oct 30 07:29:14.407209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.407200    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\") pod \"583372e5-9c7c-4e43-874f-91e94cf652ad\" (UID: \"583372e5-9c7c-4e43-874f-91e94cf652ad\") "
Oct 30 07:29:14.407422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.407392    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca" pod "583372e5-9c7c-4e43-874f-91e94cf652ad" (UID: "583372e5-9c7c-4e43-874f-91e94cf652ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:29:14.407913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.407886    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:29:14.409414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.409388    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca" pod "583372e5-9c7c-4e43-874f-91e94cf652ad" (UID: "583372e5-9c7c-4e43-874f-91e94cf652ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:29:14.409571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.409553    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px" pod "583372e5-9c7c-4e43-874f-91e94cf652ad" (UID: "583372e5-9c7c-4e43-874f-91e94cf652ad"). InnerVolumeSpecName "kube-api-access-6n9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:14.509284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.509248    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9px\" (UniqueName: \"kubernetes.io/projected/583372e5-9c7c-4e43-874f-91e94cf652ad-kube-api-access-6n9px\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:29:14.509284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.509283    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583372e5-9c7c-4e43-874f-91e94cf652ad-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:29:14.738650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:14.738552    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:15.004432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.004338    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc" event={"ID":"583372e5-9c7c-4e43-874f-91e94cf652ad","Type":"ContainerDied","Data":"11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26"}
Oct 30 07:29:15.004432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.004400    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d99da88f3cae0c67270ff3366c7c1b88be93d882a9c8d0215b1e1491c94e26"
Oct 30 07:29:15.004432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.004351    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"
Oct 30 07:29:15.032595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.032558    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"]
Oct 30 07:29:15.048515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.048473    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0860cd5-d9cdj"]
Oct 30 07:29:15.524545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.524503    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:15.686036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.685997    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:15.692036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:15.692007    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c706abed-1425-4ef9-bc1f-3d3afc005a70" path="/var/lib/kubelet/pods/c706abed-1425-4ef9-bc1f-3d3afc005a70/volumes"
Oct 30 07:29:16.736660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:16.736620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:18.604412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:18.604367    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:20.031681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:20.031646    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"]
Oct 30 07:29:20.043243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:20.043211    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9898b62-2cc7r"]
Oct 30 07:29:20.381995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:20.381953    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:20.966405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:20.966369    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:29:21.115005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:21.114965    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:21.259821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:21.259669    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:21.744804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:21.744764    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0a2639-ceca-4525-8161-245aa10231e5" path="/var/lib/kubelet/pods/cd0a2639-ceca-4525-8161-245aa10231e5/volumes"
Oct 30 07:29:22.255179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:22.255139    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:24.153167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:24.153120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:25.703255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:25.703200    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:25.967049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:25.966940    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerName="payload-processing" containerID="cri-o://b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c" gracePeriod=25
Oct 30 07:29:26.165316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.164516    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"]
Oct 30 07:29:26.514526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.513085    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4d3e92-mb7ct"]
Oct 30 07:29:26.517590 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:29:26.516730    6738 logging.go:55] [core] [Channel #4038 SubChannel #4039]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.27:50052", ServerName: "10.133.1.27:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.27:50052: connect: connection refused"
Oct 30 07:29:26.520162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.519997    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:26.650383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.634675    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:26.759169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.757783    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"]
Oct 30 07:29:26.771812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.769936    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f84d9e-kwhfc"]
Oct 30 07:29:26.798345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.798317    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:29:26.932677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.932641    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\" (UniqueName: \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\") pod \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") "
Oct 30 07:29:26.932939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.932923    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\") pod \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\" (UID: \"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d\") "
Oct 30 07:29:26.933343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.933319    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume" pod "9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" (UID: "9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:29:26.933504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.933491    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:29:26.934854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.934828    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb" pod "9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" (UID: "9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d"). InnerVolumeSpecName "kube-api-access-8t2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:26.979879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:26.979835    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:27.058370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.040168    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2sb\" (UniqueName: \"kubernetes.io/projected/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d-kube-api-access-8t2sb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:29:27.187067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.187025    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:27.213011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.212976    6738 generic.go:356] "Generic (PLEG): container finished" podID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerID="b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c" exitCode=0
Oct 30 07:29:27.213221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.213017    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"
Oct 30 07:29:27.213221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.213063    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" event={"ID":"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d","Type":"ContainerDied","Data":"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"}
Oct 30 07:29:27.213221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.213115    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" event={"ID":"9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d","Type":"ContainerDied","Data":"da607d846974fe396f804c874cbaf64d9c37b501bee4a80da16b5728932f8c41"}
Oct 30 07:29:27.213221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.213136    6738 scope.go:122] "RemoveContainer" containerID="b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"
Oct 30 07:29:27.225561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.225540    6738 scope.go:122] "RemoveContainer" containerID="b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"
Oct 30 07:29:27.225861 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:29:27.225841    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c\": container with ID starting with b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c not found: ID does not exist" containerID="b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"
Oct 30 07:29:27.225912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.225871    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c"} err="failed to get container status \"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c\": rpc error: code = NotFound desc = could not find container \"b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c\": container with ID starting with b9f63f79d9eb7fcd549ef00224ab9f3f7929b1eaa22cd8a95d342da08f2fd28c not found: ID does not exist"
Oct 30 07:29:27.267477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.267437    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:29:27.279163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.279132    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2"]
Oct 30 07:29:27.515227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.515189    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-f84d9e-766cb565f8-xg2j2" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.27:50052\" within 1s: context deadline exceeded"
Oct 30 07:29:27.691987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.691953    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583372e5-9c7c-4e43-874f-91e94cf652ad" path="/var/lib/kubelet/pods/583372e5-9c7c-4e43-874f-91e94cf652ad/volumes"
Oct 30 07:29:27.692479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.692464    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" path="/var/lib/kubelet/pods/9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d/volumes"
Oct 30 07:29:27.692915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:27.692900    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cddaaf8c-07ce-490e-9fe2-47c255578dd7" path="/var/lib/kubelet/pods/cddaaf8c-07ce-490e-9fe2-47c255578dd7/volumes"
Oct 30 07:29:28.065757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.065723    6738 scope.go:122] "RemoveContainer" containerID="5c91fe7b706116968dba21268be4f4662c83544381bb80a865aa0a8b283f2821"
Oct 30 07:29:28.076934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.076911    6738 scope.go:122] "RemoveContainer" containerID="8904a40b968ef684ad77a86dcb69d641b92ffb35d49f9e8e9962f77f8093e531"
Oct 30 07:29:28.087474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.087450    6738 scope.go:122] "RemoveContainer" containerID="74ee9ba7cb4b39605e18fa7d85c9916deb7872115b4496c219ed598325f4d71c"
Oct 30 07:29:28.097777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.097752    6738 scope.go:122] "RemoveContainer" containerID="51b19940eee9bfbf2f6163274a0f8d2cb6bc3487d593b1b65968d96191e48059"
Oct 30 07:29:28.107944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.107917    6738 scope.go:122] "RemoveContainer" containerID="40fefe15a955aaf22c4fee287fd8e036efbca4b50742636f7b293f308be6f3d7"
Oct 30 07:29:28.118674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:28.118654    6738 scope.go:122] "RemoveContainer" containerID="54dd60bfbb7f829e395bf60c4e82ea127c357d206e4d6bef75fc19d7037d20b5"
Oct 30 07:29:29.220320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:29.220275    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:31.123860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:31.123820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:31.438743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:31.438648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:33.243831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:33.243791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:33.613352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:33.613218    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:33.732368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:33.728732    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:33.988802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:33.988767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:35.211672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:35.211634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:37.032533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:37.032501    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"]
Oct 30 07:29:37.044412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:37.044383    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d76536-24n8g"]
Oct 30 07:29:37.045625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:37.045587    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:37.520988 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:29:37.520948    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:29:37.692501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:37.692470    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2349be3-5aea-4165-8fa7-afd82b46caf4" path="/var/lib/kubelet/pods/e2349be3-5aea-4165-8fa7-afd82b46caf4/volumes"
Oct 30 07:29:38.040420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:38.040379    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"]
Oct 30 07:29:38.048154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:38.048117    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60a44ed-pb6xj"]
Oct 30 07:29:38.950423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:38.950376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:39.060477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:39.058346    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"]
Oct 30 07:29:39.066094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:39.066056    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ac8b76-w8mrf"]
Oct 30 07:29:39.692219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:39.692182    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445cec8f-7035-4bca-a3bd-4a064790caa2" path="/var/lib/kubelet/pods/445cec8f-7035-4bca-a3bd-4a064790caa2/volumes"
Oct 30 07:29:39.692802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:39.692786    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d84550-81b0-4945-a6a3-df70b8cd4f85" path="/var/lib/kubelet/pods/b8d84550-81b0-4945-a6a3-df70b8cd4f85/volumes"
Oct 30 07:29:40.589614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:40.589520    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:40.860564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:40.860460    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:41.153288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:41.153243    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:41.307813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:41.307779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:42.840388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:42.840348    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:44.036160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:44.036124    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"]
Oct 30 07:29:44.048476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:44.048442    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-638c7b92-2k2vk"]
Oct 30 07:29:44.876891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:44.876850    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:45.562922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:45.562881    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:45.679835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:45.679795    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:45.692920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:45.692888    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0" path="/var/lib/kubelet/pods/9fc39cf6-3030-4e3d-a8e8-877dc5c88ac0/volumes"
Oct 30 07:29:46.811733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:46.811696    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:47.031941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.031906    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"]
Oct 30 07:29:47.043881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.043845    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07964a49-5jfv7"]
Oct 30 07:29:47.331083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.331040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:47.477366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.477327    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:47.486507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.486481    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 07:29:47.487266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487249    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerName="payload-processing"
Oct 30 07:29:47.487324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487268    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerName="payload-processing"
Oct 30 07:29:47.487324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487321    6738 cpu_manager.go:414] "containerMap: removing container" podUID="583372e5-9c7c-4e43-874f-91e94cf652ad" containerName="revoke-keys"
Oct 30 07:29:47.487405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487327    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583372e5-9c7c-4e43-874f-91e94cf652ad" containerName="revoke-keys"
Oct 30 07:29:47.487441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487435    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbbcb7b-df22-4aca-a8e5-eb472c6ed30d" containerName="payload-processing"
Oct 30 07:29:47.487476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.487449    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="583372e5-9c7c-4e43-874f-91e94cf652ad" containerName="revoke-keys"
Oct 30 07:29:47.491472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.491453    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.510036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.510006    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 07:29:47.517783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.517744    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a475d6\""
Oct 30 07:29:47.517783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.517757    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a475d6-dockercfg-zjzph\""
Oct 30 07:29:47.566854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.566814    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.567259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.567231    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twb64\" (UniqueName: \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.672321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.671927    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.672321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.672108    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twb64\" (UniqueName: \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.689319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.673688    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.736333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.729574    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twb64\" (UniqueName: \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\") pod \"payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.809667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.809628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:47.877174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:47.877133    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dceec2b0-1460-45f6-8f03-859720056a6e" path="/var/lib/kubelet/pods/dceec2b0-1460-45f6-8f03-859720056a6e/volumes"
Oct 30 07:29:48.265374 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:29:48.265341    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07b0829_804f_4ef2_929e_482c02ca641c.slice/crio-32e92936f52236c5f9868efed1c6b43231ee47d0a63bd8f034d5fb43e93e3e5b WatchSource:0}: Error finding container 32e92936f52236c5f9868efed1c6b43231ee47d0a63bd8f034d5fb43e93e3e5b: Status 404 returned error can't find the container with id 32e92936f52236c5f9868efed1c6b43231ee47d0a63bd8f034d5fb43e93e3e5b
Oct 30 07:29:48.265519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.265422    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 07:29:48.556012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.555923    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" event={"ID":"a07b0829-804f-4ef2-929e-482c02ca641c","Type":"ContainerStarted","Data":"4e6e98850a8f57a284cb4ba687ea4a89521d134f819764b1c7246ce55eca9d4a"}
Oct 30 07:29:48.556012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.555982    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" event={"ID":"a07b0829-804f-4ef2-929e-482c02ca641c","Type":"ContainerStarted","Data":"32e92936f52236c5f9868efed1c6b43231ee47d0a63bd8f034d5fb43e93e3e5b"}
Oct 30 07:29:48.556012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.556013    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:29:48.603562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.603509    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" podStartSLOduration=1.6034960379999998 podStartE2EDuration="1.603496038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:48.599718016 +0000 UTC m=+1949.032481908" watchObservedRunningTime="2026-10-30 07:29:48.603496038 +0000 UTC m=+1949.036259931"
Oct 30 07:29:48.822373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:48.822237    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:50.747880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:50.746090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:52.571254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:52.571209    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:53.031115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:53.031067    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"]
Oct 30 07:29:53.043667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:53.043632    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3071736-2kzsm"]
Oct 30 07:29:53.692002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:53.691969    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8e635e-9d58-43da-9719-554636c934eb" path="/var/lib/kubelet/pods/2a8e635e-9d58-43da-9719-554636c934eb/volumes"
Oct 30 07:29:54.426073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:54.426041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:56.735836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:56.735792    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:58.522871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:58.522826    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:29:59.567918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:29:59.567886    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 07:30:00.159370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.159330    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:00.164520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.164490    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.195550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.195503    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:00.201177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.201145    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:30:00.202810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.202786    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:00.204664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.204635    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.213218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.213170    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.222347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.222177    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq844\" (UniqueName: \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.222722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.222643    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.236885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.235550    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:00.242337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.242282    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.252793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.252758    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:00.257807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.257770    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:00.266649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.266614    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:00.276143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.276107    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:00.281470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.281419    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:00.295718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.295685    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.301745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.301717    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq844\" (UniqueName: \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325151    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwhc\" (UniqueName: \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325181    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325222    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrn9\" (UniqueName: \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4cf\" (UniqueName: \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325387    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.325437    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.326213    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.326516    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49gm\" (UniqueName: \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.326674    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.326777    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.327676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.327595    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:00.328622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.328452    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.333928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.332742    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:00.334091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.334010    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.338079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.338056    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:00.338227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.338094    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:00.338227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.338130    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:00.338227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.338154    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:00.338227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.338174    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.362661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.362629    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq844\" (UniqueName: \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.428581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428489    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.428581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428541    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6td6\" (UniqueName: \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.428804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428612    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.428804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428659    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.428804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428696    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwhc\" (UniqueName: \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.428804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428717    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.428804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrn9\" (UniqueName: \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.429027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428819    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4cf\" (UniqueName: \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.429027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.428973    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.429134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429026    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskz9\" (UniqueName: \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.429134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfdn\" (UniqueName: \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.429253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429227    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49gm\" (UniqueName: \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.429362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429343    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.429482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.429570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429501    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.429749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429725    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.430010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.429990    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.430119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.430103    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.454353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.454269    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrn9\" (UniqueName: \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.454513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.454395    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwhc\" (UniqueName: \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.454513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.454396    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4cf\" (UniqueName: \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.458664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.458631    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49gm\" (UniqueName: \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.516327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.504835    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:00.522461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.522319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.546997    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.547066    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6td6\" (UniqueName: \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.547112    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.547144    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.547337    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskz9\" (UniqueName: \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.547452    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfdn\" (UniqueName: \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.548934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.554930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.549903    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.561477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.561440    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.566680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.566646    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:00.579330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.579277    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:00.599280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.599245    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:00.609752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.608240    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6td6\" (UniqueName: \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.610039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.609928    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskz9\" (UniqueName: \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:00.610039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.609986    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfdn\" (UniqueName: \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.626465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.626434    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:00.652541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.652147    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:00.695731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.695454    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:00.696186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:00.696121    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:01.047781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.047679    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"]
Oct 30 07:30:01.058035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.057977    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"]
Oct 30 07:30:01.061487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.061451    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c87e9-hbjzg"]
Oct 30 07:30:01.067380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.067343    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525d33eb-4vh4v"]
Oct 30 07:30:01.134886 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.134840    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d91bde4_c915_4bb5_991c_5a0884d0a207.slice/crio-c88c6736c62ca244c78ea4c28567c8ee228239cffda2a892ebb8a370f96dc1a5 WatchSource:0}: Error finding container c88c6736c62ca244c78ea4c28567c8ee228239cffda2a892ebb8a370f96dc1a5: Status 404 returned error can't find the container with id c88c6736c62ca244c78ea4c28567c8ee228239cffda2a892ebb8a370f96dc1a5
Oct 30 07:30:01.135169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.135144    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:01.151339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.151262    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:01.159768 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.159733    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230a268c_877c_4dde_9dbc_0928c029afc9.slice/crio-2630defbe105a5e4f098e6aedb0243418301ae698e260ff5d0dbc06236771e85 WatchSource:0}: Error finding container 2630defbe105a5e4f098e6aedb0243418301ae698e260ff5d0dbc06236771e85: Status 404 returned error can't find the container with id 2630defbe105a5e4f098e6aedb0243418301ae698e260ff5d0dbc06236771e85
Oct 30 07:30:01.160778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.160746    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:01.181645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.181618    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:01.185674 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.185576    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7becc556_7eff_4b9f_9bd7_4041fd30c043.slice/crio-03c14fc7e34d4233377e3cf76ed44eda2a551fed21f11baa3874dda19fdffb0b WatchSource:0}: Error finding container 03c14fc7e34d4233377e3cf76ed44eda2a551fed21f11baa3874dda19fdffb0b: Status 404 returned error can't find the container with id 03c14fc7e34d4233377e3cf76ed44eda2a551fed21f11baa3874dda19fdffb0b
Oct 30 07:30:01.199379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.199349    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:01.244234 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.244198    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8cce03_fd94_4e26_8a24_de16b92cf71d.slice/crio-16c0cc02c0e764b45025f18927589b7b6547137abdcfa5f86dc857b5b65936ac WatchSource:0}: Error finding container 16c0cc02c0e764b45025f18927589b7b6547137abdcfa5f86dc857b5b65936ac: Status 404 returned error can't find the container with id 16c0cc02c0e764b45025f18927589b7b6547137abdcfa5f86dc857b5b65936ac
Oct 30 07:30:01.245665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.245637    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:01.253876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.253850    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:01.259754 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.259725    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d448b_42b0_43bb_9665_7997087515b8.slice/crio-c9c0ff54e569b8e07e68ded12307a8b2ddfd521e170aca48fb86745ebc09f3b0 WatchSource:0}: Error finding container c9c0ff54e569b8e07e68ded12307a8b2ddfd521e170aca48fb86745ebc09f3b0: Status 404 returned error can't find the container with id c9c0ff54e569b8e07e68ded12307a8b2ddfd521e170aca48fb86745ebc09f3b0
Oct 30 07:30:01.317204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.317171    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:01.375026 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:01.374992    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64d6681_ac3f_4e97_a2f0_1483bd566653.slice/crio-61f69178a18e4fa3585af4ad4110642cc3fb9122762c8ed3d8c3185d7da6e86e WatchSource:0}: Error finding container 61f69178a18e4fa3585af4ad4110642cc3fb9122762c8ed3d8c3185d7da6e86e: Status 404 returned error can't find the container with id 61f69178a18e4fa3585af4ad4110642cc3fb9122762c8ed3d8c3185d7da6e86e
Oct 30 07:30:01.693826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.693794    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370bba0e-f2cf-42aa-a5c4-e62ad213d627" path="/var/lib/kubelet/pods/370bba0e-f2cf-42aa-a5c4-e62ad213d627/volumes"
Oct 30 07:30:01.694379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.694363    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54" path="/var/lib/kubelet/pods/be76e8dc-0fb5-41cd-8b4a-5f2411e9cc54/volumes"
Oct 30 07:30:01.918050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.918012    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerStarted","Data":"61f69178a18e4fa3585af4ad4110642cc3fb9122762c8ed3d8c3185d7da6e86e"}
Oct 30 07:30:01.919935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.919903    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerStarted","Data":"06f93f8e80f9109b9e3758a76c50550204171bd6c2a6c99ce64a5fc7ede387b7"}
Oct 30 07:30:01.920049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.919961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerStarted","Data":"ca190a8a64554db585e9101868f1ed0637d83341f920f543feb5d0593808c6c3"}
Oct 30 07:30:01.921750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.921724    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerStarted","Data":"880de6cd645b2a635e26f9646450721a48e1a1449b9efc410499052cf9acb057"}
Oct 30 07:30:01.921876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.921774    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerStarted","Data":"16c0cc02c0e764b45025f18927589b7b6547137abdcfa5f86dc857b5b65936ac"}
Oct 30 07:30:01.923515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.923486    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerStarted","Data":"a00def00bbba15443b4da3780f6dc0125acb412224982bdadd53967bed4bba13"}
Oct 30 07:30:01.923626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.923538    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerStarted","Data":"03c14fc7e34d4233377e3cf76ed44eda2a551fed21f11baa3874dda19fdffb0b"}
Oct 30 07:30:01.925257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.925234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerStarted","Data":"3b94b13f423f5fe8b097da5c2a13fef66a3790efe7d24a925eab07f9a03a9d35"}
Oct 30 07:30:01.925369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.925272    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerStarted","Data":"c9c0ff54e569b8e07e68ded12307a8b2ddfd521e170aca48fb86745ebc09f3b0"}
Oct 30 07:30:01.926874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.926845    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerStarted","Data":"f111b6d4b14910a27b785e23032835a6915051cfa12415144bdee7e911465fce"}
Oct 30 07:30:01.926983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.926892    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerStarted","Data":"2630defbe105a5e4f098e6aedb0243418301ae698e260ff5d0dbc06236771e85"}
Oct 30 07:30:01.928427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.928402    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerStarted","Data":"891a1be0bb0e8c5c0e6a5e60821a3cbbdcfede27faa67395891576b1f5d33512"}
Oct 30 07:30:01.928528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.928441    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerStarted","Data":"c88c6736c62ca244c78ea4c28567c8ee228239cffda2a892ebb8a370f96dc1a5"}
Oct 30 07:30:01.930132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.930112    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerStarted","Data":"f1f95339a28b05e6d6e029aa31acdd94882696090f574fd13adb0bb64a161db1"}
Oct 30 07:30:01.930239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.930145    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerStarted","Data":"a5220eafd520251f2e9c4b5f85b9fffafd13f4d8a9e535448b1bd000656264d7"}
Oct 30 07:30:01.976551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:01.976492    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" podStartSLOduration=1.976471834 podStartE2EDuration="1.976471834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.973153363 +0000 UTC m=+1962.405917248" watchObservedRunningTime="2026-10-30 07:30:01.976471834 +0000 UTC m=+1962.409235726"
Oct 30 07:30:02.018937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.018858    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" podStartSLOduration=2.018840901 podStartE2EDuration="2.018840901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.01875401 +0000 UTC m=+1962.451517903" watchObservedRunningTime="2026-10-30 07:30:02.018840901 +0000 UTC m=+1962.451604794"
Oct 30 07:30:02.069720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.069654    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" podStartSLOduration=2.069632785 podStartE2EDuration="2.069632785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.065853307 +0000 UTC m=+1962.498617201" watchObservedRunningTime="2026-10-30 07:30:02.069632785 +0000 UTC m=+1962.502396679"
Oct 30 07:30:02.114633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.114580    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" podStartSLOduration=2.1145661159999998 podStartE2EDuration="2.114566116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.110644294 +0000 UTC m=+1962.543408187" watchObservedRunningTime="2026-10-30 07:30:02.114566116 +0000 UTC m=+1962.547330009"
Oct 30 07:30:02.155632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.155576    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" podStartSLOduration=2.1555626549999998 podStartE2EDuration="2.155562655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.150896094 +0000 UTC m=+1962.583659986" watchObservedRunningTime="2026-10-30 07:30:02.155562655 +0000 UTC m=+1962.588326547"
Oct 30 07:30:02.197568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.197501    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" podStartSLOduration=2.197486764 podStartE2EDuration="2.197486764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.193734881 +0000 UTC m=+1962.626498775" watchObservedRunningTime="2026-10-30 07:30:02.197486764 +0000 UTC m=+1962.630250657"
Oct 30 07:30:02.242198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.242086    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" podStartSLOduration=2.242069517 podStartE2EDuration="2.242069517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.237745664 +0000 UTC m=+1962.670509554" watchObservedRunningTime="2026-10-30 07:30:02.242069517 +0000 UTC m=+1962.674833446"
Oct 30 07:30:02.615819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.615698    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:02.942030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.941990    6738 generic.go:356] "Generic (PLEG): container finished" podID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerID="06f93f8e80f9109b9e3758a76c50550204171bd6c2a6c99ce64a5fc7ede387b7" exitCode=60
Oct 30 07:30:02.942542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.942028    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerDied","Data":"06f93f8e80f9109b9e3758a76c50550204171bd6c2a6c99ce64a5fc7ede387b7"}
Oct 30 07:30:02.942542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.942514    6738 scope.go:122] "RemoveContainer" containerID="06f93f8e80f9109b9e3758a76c50550204171bd6c2a6c99ce64a5fc7ede387b7"
Oct 30 07:30:02.944066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.944044    6738 generic.go:356] "Generic (PLEG): container finished" podID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerID="880de6cd645b2a635e26f9646450721a48e1a1449b9efc410499052cf9acb057" exitCode=60
Oct 30 07:30:02.944179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.944131    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerDied","Data":"880de6cd645b2a635e26f9646450721a48e1a1449b9efc410499052cf9acb057"}
Oct 30 07:30:02.944474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.944446    6738 scope.go:122] "RemoveContainer" containerID="880de6cd645b2a635e26f9646450721a48e1a1449b9efc410499052cf9acb057"
Oct 30 07:30:02.946249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.946226    6738 generic.go:356] "Generic (PLEG): container finished" podID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerID="a00def00bbba15443b4da3780f6dc0125acb412224982bdadd53967bed4bba13" exitCode=60
Oct 30 07:30:02.946374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.946312    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerDied","Data":"a00def00bbba15443b4da3780f6dc0125acb412224982bdadd53967bed4bba13"}
Oct 30 07:30:02.946654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.946573    6738 scope.go:122] "RemoveContainer" containerID="a00def00bbba15443b4da3780f6dc0125acb412224982bdadd53967bed4bba13"
Oct 30 07:30:02.948570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.948546    6738 generic.go:356] "Generic (PLEG): container finished" podID="a29d448b-42b0-43bb-9665-7997087515b8" containerID="3b94b13f423f5fe8b097da5c2a13fef66a3790efe7d24a925eab07f9a03a9d35" exitCode=60
Oct 30 07:30:02.948651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.948624    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerDied","Data":"3b94b13f423f5fe8b097da5c2a13fef66a3790efe7d24a925eab07f9a03a9d35"}
Oct 30 07:30:02.948873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.948853    6738 scope.go:122] "RemoveContainer" containerID="3b94b13f423f5fe8b097da5c2a13fef66a3790efe7d24a925eab07f9a03a9d35"
Oct 30 07:30:02.950516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.950497    6738 generic.go:356] "Generic (PLEG): container finished" podID="230a268c-877c-4dde-9dbc-0928c029afc9" containerID="f111b6d4b14910a27b785e23032835a6915051cfa12415144bdee7e911465fce" exitCode=60
Oct 30 07:30:02.950608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.950562    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerDied","Data":"f111b6d4b14910a27b785e23032835a6915051cfa12415144bdee7e911465fce"}
Oct 30 07:30:02.950822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.950803    6738 scope.go:122] "RemoveContainer" containerID="f111b6d4b14910a27b785e23032835a6915051cfa12415144bdee7e911465fce"
Oct 30 07:30:02.952506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.952484    6738 generic.go:356] "Generic (PLEG): container finished" podID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerID="891a1be0bb0e8c5c0e6a5e60821a3cbbdcfede27faa67395891576b1f5d33512" exitCode=60
Oct 30 07:30:02.952621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.952568    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerDied","Data":"891a1be0bb0e8c5c0e6a5e60821a3cbbdcfede27faa67395891576b1f5d33512"}
Oct 30 07:30:02.952796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.952779    6738 scope.go:122] "RemoveContainer" containerID="891a1be0bb0e8c5c0e6a5e60821a3cbbdcfede27faa67395891576b1f5d33512"
Oct 30 07:30:02.954420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.954396    6738 generic.go:356] "Generic (PLEG): container finished" podID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerID="f1f95339a28b05e6d6e029aa31acdd94882696090f574fd13adb0bb64a161db1" exitCode=60
Oct 30 07:30:02.954522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.954477    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerDied","Data":"f1f95339a28b05e6d6e029aa31acdd94882696090f574fd13adb0bb64a161db1"}
Oct 30 07:30:02.954706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.954691    6738 scope.go:122] "RemoveContainer" containerID="f1f95339a28b05e6d6e029aa31acdd94882696090f574fd13adb0bb64a161db1"
Oct 30 07:30:02.957406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.957269    6738 generic.go:356] "Generic (PLEG): container finished" podID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerID="f254b91da512b98131859ec4f0dd69e5b83900dd6bc10b1c127d4d3b0a26372f" exitCode=60
Oct 30 07:30:02.957406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.957309    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerDied","Data":"f254b91da512b98131859ec4f0dd69e5b83900dd6bc10b1c127d4d3b0a26372f"}
Oct 30 07:30:02.957663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:02.957641    6738 scope.go:122] "RemoveContainer" containerID="f254b91da512b98131859ec4f0dd69e5b83900dd6bc10b1c127d4d3b0a26372f"
Oct 30 07:30:03.066201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.066173    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"]
Oct 30 07:30:03.079284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.079249    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a029f64-62789"]
Oct 30 07:30:03.694429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.694357    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2c2274-b173-457e-985c-6d5b9f871a7e" path="/var/lib/kubelet/pods/3b2c2274-b173-457e-985c-6d5b9f871a7e/volumes"
Oct 30 07:30:03.970547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.970427    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerStarted","Data":"5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"}
Oct 30 07:30:03.973863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.973833    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerStarted","Data":"758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"}
Oct 30 07:30:03.976211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.976182    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerStarted","Data":"29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"}
Oct 30 07:30:03.978267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.978241    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerStarted","Data":"230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"}
Oct 30 07:30:03.980398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.980370    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerStarted","Data":"a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"}
Oct 30 07:30:03.982857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.982832    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerStarted","Data":"ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"}
Oct 30 07:30:03.985064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.985037    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerStarted","Data":"1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"}
Oct 30 07:30:03.987726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:03.987701    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerStarted","Data":"b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"}
Oct 30 07:30:04.085936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.085878    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" podStartSLOduration=4.085864191 podStartE2EDuration="4.085864191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:04.082250711 +0000 UTC m=+1964.515014603" watchObservedRunningTime="2026-10-30 07:30:04.085864191 +0000 UTC m=+1964.518628084"
Oct 30 07:30:04.645106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.645067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:04.998394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.998271    6738 generic.go:356] "Generic (PLEG): container finished" podID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerID="230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9" exitCode=60
Oct 30 07:30:04.998394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.998348    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerDied","Data":"230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"}
Oct 30 07:30:04.998930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.998408    6738 scope.go:122] "RemoveContainer" containerID="f254b91da512b98131859ec4f0dd69e5b83900dd6bc10b1c127d4d3b0a26372f"
Oct 30 07:30:04.998930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:04.998797    6738 scope.go:122] "RemoveContainer" containerID="230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"
Oct 30 07:30:04.999269 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:04.999238    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg_odh-ai-gateway-infra(a64d6681-ac3f-4e97-a2f0-1483bd566653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653"
Oct 30 07:30:05.000636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.000613    6738 generic.go:356] "Generic (PLEG): container finished" podID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerID="a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0" exitCode=60
Oct 30 07:30:05.000765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.000699    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerDied","Data":"a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"}
Oct 30 07:30:05.001003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.000990    6738 scope.go:122] "RemoveContainer" containerID="a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"
Oct 30 07:30:05.001501 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.001475    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf_odh-ai-gateway-infra(0871b9f2-fc43-473b-8e0c-ebe9eb897330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330"
Oct 30 07:30:05.002733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.002712    6738 generic.go:356] "Generic (PLEG): container finished" podID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerID="ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f" exitCode=60
Oct 30 07:30:05.002857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.002795    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerDied","Data":"ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"}
Oct 30 07:30:05.003064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.003049    6738 scope.go:122] "RemoveContainer" containerID="ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"
Oct 30 07:30:05.003501 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.003473    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9_odh-ai-gateway-infra(3a8cce03-fd94-4e26-8a24-de16b92cf71d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d"
Oct 30 07:30:05.005216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.005190    6738 generic.go:356] "Generic (PLEG): container finished" podID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerID="1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48" exitCode=60
Oct 30 07:30:05.005355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.005284    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerDied","Data":"1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"}
Oct 30 07:30:05.005629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.005605    6738 scope.go:122] "RemoveContainer" containerID="1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"
Oct 30 07:30:05.006053 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.006032    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2_odh-ai-gateway-infra(7becc556-7eff-4b9f-9bd7-4041fd30c043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043"
Oct 30 07:30:05.007919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.007895    6738 generic.go:356] "Generic (PLEG): container finished" podID="a29d448b-42b0-43bb-9665-7997087515b8" containerID="b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573" exitCode=60
Oct 30 07:30:05.008037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.008012    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerDied","Data":"b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"}
Oct 30 07:30:05.008225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.008211    6738 scope.go:122] "RemoveContainer" containerID="b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"
Oct 30 07:30:05.008593 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.008570    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx_odh-ai-gateway-infra(a29d448b-42b0-43bb-9665-7997087515b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" podUID="a29d448b-42b0-43bb-9665-7997087515b8"
Oct 30 07:30:05.010062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.010043    6738 generic.go:356] "Generic (PLEG): container finished" podID="230a268c-877c-4dde-9dbc-0928c029afc9" containerID="5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680" exitCode=60
Oct 30 07:30:05.010164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.010069    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerDied","Data":"5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"}
Oct 30 07:30:05.010499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.010477    6738 scope.go:122] "RemoveContainer" containerID="5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"
Oct 30 07:30:05.010879 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.010857    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8_odh-ai-gateway-infra(230a268c-877c-4dde-9dbc-0928c029afc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" podUID="230a268c-877c-4dde-9dbc-0928c029afc9"
Oct 30 07:30:05.012805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.012659    6738 generic.go:356] "Generic (PLEG): container finished" podID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerID="758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6" exitCode=60
Oct 30 07:30:05.012805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.012716    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerDied","Data":"758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"}
Oct 30 07:30:05.013053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.013034    6738 scope.go:122] "RemoveContainer" containerID="758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"
Oct 30 07:30:05.013377 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.013358    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8_odh-ai-gateway-infra(4d91bde4-c915-4bb5-991c-5a0884d0a207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207"
Oct 30 07:30:05.015463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.015440    6738 generic.go:356] "Generic (PLEG): container finished" podID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerID="29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9" exitCode=60
Oct 30 07:30:05.015585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.015528    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerDied","Data":"29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"}
Oct 30 07:30:05.015769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.015751    6738 scope.go:122] "RemoveContainer" containerID="29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"
Oct 30 07:30:05.015860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.015839    6738 scope.go:122] "RemoveContainer" containerID="06f93f8e80f9109b9e3758a76c50550204171bd6c2a6c99ce64a5fc7ede387b7"
Oct 30 07:30:05.016124 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:05.016103    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v_odh-ai-gateway-infra(e7e60706-5eb9-47c9-8611-a8dd8202ec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b"
Oct 30 07:30:05.031758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.031735    6738 scope.go:122] "RemoveContainer" containerID="880de6cd645b2a635e26f9646450721a48e1a1449b9efc410499052cf9acb057"
Oct 30 07:30:05.048764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.048737    6738 scope.go:122] "RemoveContainer" containerID="a00def00bbba15443b4da3780f6dc0125acb412224982bdadd53967bed4bba13"
Oct 30 07:30:05.071972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.071934    6738 scope.go:122] "RemoveContainer" containerID="3b94b13f423f5fe8b097da5c2a13fef66a3790efe7d24a925eab07f9a03a9d35"
Oct 30 07:30:05.087711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.087678    6738 scope.go:122] "RemoveContainer" containerID="f111b6d4b14910a27b785e23032835a6915051cfa12415144bdee7e911465fce"
Oct 30 07:30:05.106768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.106737    6738 scope.go:122] "RemoveContainer" containerID="891a1be0bb0e8c5c0e6a5e60821a3cbbdcfede27faa67395891576b1f5d33512"
Oct 30 07:30:05.125305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:05.125273    6738 scope.go:122] "RemoveContainer" containerID="f1f95339a28b05e6d6e029aa31acdd94882696090f574fd13adb0bb64a161db1"
Oct 30 07:30:06.028282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.028250    6738 scope.go:122] "RemoveContainer" containerID="b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"
Oct 30 07:30:06.028778 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.028731    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx_odh-ai-gateway-infra(a29d448b-42b0-43bb-9665-7997087515b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" podUID="a29d448b-42b0-43bb-9665-7997087515b8"
Oct 30 07:30:06.030329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.030270    6738 scope.go:122] "RemoveContainer" containerID="5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"
Oct 30 07:30:06.030764 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.030724    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8_odh-ai-gateway-infra(230a268c-877c-4dde-9dbc-0928c029afc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" podUID="230a268c-877c-4dde-9dbc-0928c029afc9"
Oct 30 07:30:06.031917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.031896    6738 scope.go:122] "RemoveContainer" containerID="758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"
Oct 30 07:30:06.032269 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.032248    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8_odh-ai-gateway-infra(4d91bde4-c915-4bb5-991c-5a0884d0a207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207"
Oct 30 07:30:06.033476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.033457    6738 scope.go:122] "RemoveContainer" containerID="29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"
Oct 30 07:30:06.033798 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.033778    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v_odh-ai-gateway-infra(e7e60706-5eb9-47c9-8611-a8dd8202ec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b"
Oct 30 07:30:06.035107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.035086    6738 scope.go:122] "RemoveContainer" containerID="230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"
Oct 30 07:30:06.035477 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.035453    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg_odh-ai-gateway-infra(a64d6681-ac3f-4e97-a2f0-1483bd566653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653"
Oct 30 07:30:06.036696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.036680    6738 scope.go:122] "RemoveContainer" containerID="a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"
Oct 30 07:30:06.037009 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.036987    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf_odh-ai-gateway-infra(0871b9f2-fc43-473b-8e0c-ebe9eb897330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330"
Oct 30 07:30:06.038257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.038240    6738 scope.go:122] "RemoveContainer" containerID="ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"
Oct 30 07:30:06.038564 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.038542    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9_odh-ai-gateway-infra(3a8cce03-fd94-4e26-8a24-de16b92cf71d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d"
Oct 30 07:30:06.039980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.039963    6738 scope.go:122] "RemoveContainer" containerID="1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"
Oct 30 07:30:06.040207 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:06.040191    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2_odh-ai-gateway-infra(7becc556-7eff-4b9f-9bd7-4041fd30c043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043"
Oct 30 07:30:06.718933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:06.718895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:07.522589 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:07.522552    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:30:08.523163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:08.523123    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:09.037201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:09.037166    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"]
Oct 30 07:30:09.049380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:09.049347    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deffd453-fxjk6"]
Oct 30 07:30:09.692596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:09.692562    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33380a5-dac3-4efb-90de-7de883744c04" path="/var/lib/kubelet/pods/f33380a5-dac3-4efb-90de-7de883744c04/volumes"
Oct 30 07:30:10.354221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:10.354182    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:12.195920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:12.195873    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:12.330321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:12.328029    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:12.478833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:12.478724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:13.033734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:13.033702    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"]
Oct 30 07:30:13.046224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:13.046190    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e2cac9-582xr"]
Oct 30 07:30:13.692887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:13.692849    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a33b917-2d25-4698-a379-92306d95c6bb" path="/var/lib/kubelet/pods/5a33b917-2d25-4698-a379-92306d95c6bb/volumes"
Oct 30 07:30:14.243411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:14.243371    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:14.683073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:14.683042    6738 scope.go:122] "RemoveContainer" containerID="230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"
Oct 30 07:30:15.177610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:15.177577    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerStarted","Data":"d827c55731259e944de19933b56be92c8131c7049671724f8f649bef197fb1bd"}
Oct 30 07:30:15.683865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:15.683829    6738 scope.go:122] "RemoveContainer" containerID="a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"
Oct 30 07:30:15.684037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:15.683956    6738 scope.go:122] "RemoveContainer" containerID="b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"
Oct 30 07:30:16.189967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.189933    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerStarted","Data":"71bedcd5f19a217371b802f47a974610271cc8672229e5723af31d2c6fb1e4cc"}
Oct 30 07:30:16.193273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.192812    6738 generic.go:356] "Generic (PLEG): container finished" podID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerID="d827c55731259e944de19933b56be92c8131c7049671724f8f649bef197fb1bd" exitCode=60
Oct 30 07:30:16.193273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.192851    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerDied","Data":"d827c55731259e944de19933b56be92c8131c7049671724f8f649bef197fb1bd"}
Oct 30 07:30:16.193273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.192913    6738 scope.go:122] "RemoveContainer" containerID="230f258e1509cfa1a813fa3c9431e4572e7e47db9b9b7827b8cfb090f28297d9"
Oct 30 07:30:16.193579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.193346    6738 scope.go:122] "RemoveContainer" containerID="d827c55731259e944de19933b56be92c8131c7049671724f8f649bef197fb1bd"
Oct 30 07:30:16.193794 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:16.193767    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg_odh-ai-gateway-infra(a64d6681-ac3f-4e97-a2f0-1483bd566653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653"
Oct 30 07:30:16.195216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.195190    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerStarted","Data":"4e6b119cee307b83bf857294e0493f6f02b0b70cb1c91724243cc54269c13b75"}
Oct 30 07:30:16.241279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.241246    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:16.282818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.282781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:16.684013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.683985    6738 scope.go:122] "RemoveContainer" containerID="ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"
Oct 30 07:30:16.684365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.684343    6738 scope.go:122] "RemoveContainer" containerID="5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"
Oct 30 07:30:16.684495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:16.684414    6738 scope.go:122] "RemoveContainer" containerID="29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"
Oct 30 07:30:17.207476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.207443    6738 generic.go:356] "Generic (PLEG): container finished" podID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerID="4e6b119cee307b83bf857294e0493f6f02b0b70cb1c91724243cc54269c13b75" exitCode=60
Oct 30 07:30:17.207936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.207512    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerDied","Data":"4e6b119cee307b83bf857294e0493f6f02b0b70cb1c91724243cc54269c13b75"}
Oct 30 07:30:17.207936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.207573    6738 scope.go:122] "RemoveContainer" containerID="a914c0b68c751c1ee15308a52546ea9d14fdc67b727a178ec77d8e1087023eb0"
Oct 30 07:30:17.208079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.208061    6738 scope.go:122] "RemoveContainer" containerID="4e6b119cee307b83bf857294e0493f6f02b0b70cb1c91724243cc54269c13b75"
Oct 30 07:30:17.208641 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:17.208570    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf_odh-ai-gateway-infra(0871b9f2-fc43-473b-8e0c-ebe9eb897330)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330"
Oct 30 07:30:17.210023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.210000    6738 generic.go:356] "Generic (PLEG): container finished" podID="a29d448b-42b0-43bb-9665-7997087515b8" containerID="71bedcd5f19a217371b802f47a974610271cc8672229e5723af31d2c6fb1e4cc" exitCode=60
Oct 30 07:30:17.210143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.210069    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerDied","Data":"71bedcd5f19a217371b802f47a974610271cc8672229e5723af31d2c6fb1e4cc"}
Oct 30 07:30:17.210553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.210518    6738 scope.go:122] "RemoveContainer" containerID="71bedcd5f19a217371b802f47a974610271cc8672229e5723af31d2c6fb1e4cc"
Oct 30 07:30:17.211001 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:17.210974    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx_odh-ai-gateway-infra(a29d448b-42b0-43bb-9665-7997087515b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" podUID="a29d448b-42b0-43bb-9665-7997087515b8"
Oct 30 07:30:17.212561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.212540    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerStarted","Data":"40832e52e8185c886ac6d07d28d4b9d2ed1355f4e1ec5ad7df3822c8c201f2a5"}
Oct 30 07:30:17.223948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.223925    6738 scope.go:122] "RemoveContainer" containerID="b379a1f44c6380b18fa87a7705d7183309a65da86dc1f2aa869b7ef5088da573"
Oct 30 07:30:17.246845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.246808    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:17.328756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.328718    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:17.402596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.402566    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:17.477759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.477711    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\") pod \"a64d6681-ac3f-4e97-a2f0-1483bd566653\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") "
Oct 30 07:30:17.478095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.478076    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\" (UniqueName: \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\") pod \"a64d6681-ac3f-4e97-a2f0-1483bd566653\" (UID: \"a64d6681-ac3f-4e97-a2f0-1483bd566653\") "
Oct 30 07:30:17.478220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.478106    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca" pod "a64d6681-ac3f-4e97-a2f0-1483bd566653" (UID: "a64d6681-ac3f-4e97-a2f0-1483bd566653"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:17.478706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.478687    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64d6681-ac3f-4e97-a2f0-1483bd566653-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:17.480143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.480122    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6" pod "a64d6681-ac3f-4e97-a2f0-1483bd566653" (UID: "a64d6681-ac3f-4e97-a2f0-1483bd566653"). InnerVolumeSpecName "kube-api-access-q6td6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:17.580044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.580011    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6td6\" (UniqueName: \"kubernetes.io/projected/a64d6681-ac3f-4e97-a2f0-1483bd566653-kube-api-access-q6td6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:17.682955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:17.682925    6738 scope.go:122] "RemoveContainer" containerID="1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"
Oct 30 07:30:18.223931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.223805    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:18.225226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.225202    6738 generic.go:356] "Generic (PLEG): container finished" podID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerID="40832e52e8185c886ac6d07d28d4b9d2ed1355f4e1ec5ad7df3822c8c201f2a5" exitCode=60
Oct 30 07:30:18.225380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.225273    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerDied","Data":"40832e52e8185c886ac6d07d28d4b9d2ed1355f4e1ec5ad7df3822c8c201f2a5"}
Oct 30 07:30:18.225380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.225346    6738 scope.go:122] "RemoveContainer" containerID="29ffb3a1f7559e002094a2096ae84f1b816c73a8768965e15e1fe64c128a32d9"
Oct 30 07:30:18.225804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.225787    6738 scope.go:122] "RemoveContainer" containerID="40832e52e8185c886ac6d07d28d4b9d2ed1355f4e1ec5ad7df3822c8c201f2a5"
Oct 30 07:30:18.226248 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:18.226222    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v_odh-ai-gateway-infra(e7e60706-5eb9-47c9-8611-a8dd8202ec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b"
Oct 30 07:30:18.227043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.227020    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg" event={"ID":"a64d6681-ac3f-4e97-a2f0-1483bd566653","Type":"ContainerDied","Data":"61f69178a18e4fa3585af4ad4110642cc3fb9122762c8ed3d8c3185d7da6e86e"}
Oct 30 07:30:18.227148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.227051    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"
Oct 30 07:30:18.231814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.231781    6738 generic.go:356] "Generic (PLEG): container finished" podID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerID="abd04214842f89e9bcb25625c5fd1ad509ec235c7d7b00da2dfc95265cccfdfa" exitCode=60
Oct 30 07:30:18.231940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.231848    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerDied","Data":"abd04214842f89e9bcb25625c5fd1ad509ec235c7d7b00da2dfc95265cccfdfa"}
Oct 30 07:30:18.232361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.232342    6738 scope.go:122] "RemoveContainer" containerID="abd04214842f89e9bcb25625c5fd1ad509ec235c7d7b00da2dfc95265cccfdfa"
Oct 30 07:30:18.232770 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:18.232738    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9_odh-ai-gateway-infra(3a8cce03-fd94-4e26-8a24-de16b92cf71d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d"
Oct 30 07:30:18.234128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.234103    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerStarted","Data":"a8b70f50191bcae8ad4cc81ad145e4e0b1dde5e450f60bc745bf40741b381f34"}
Oct 30 07:30:18.239030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.238984    6738 generic.go:356] "Generic (PLEG): container finished" podID="230a268c-877c-4dde-9dbc-0928c029afc9" containerID="305975dea8394a2479db441dd1688642e74ecd4d682ecd3d77893890067a45ca" exitCode=60
Oct 30 07:30:18.239148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.239091    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerDied","Data":"305975dea8394a2479db441dd1688642e74ecd4d682ecd3d77893890067a45ca"}
Oct 30 07:30:18.239544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.239526    6738 scope.go:122] "RemoveContainer" containerID="305975dea8394a2479db441dd1688642e74ecd4d682ecd3d77893890067a45ca"
Oct 30 07:30:18.239866 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:18.239846    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8_odh-ai-gateway-infra(230a268c-877c-4dde-9dbc-0928c029afc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" podUID="230a268c-877c-4dde-9dbc-0928c029afc9"
Oct 30 07:30:18.242254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.242236    6738 scope.go:122] "RemoveContainer" containerID="d827c55731259e944de19933b56be92c8131c7049671724f8f649bef197fb1bd"
Oct 30 07:30:18.288735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.286592    6738 scope.go:122] "RemoveContainer" containerID="ee26f6b4249f14a0b2dd4522710b13c67e2c5e68b0902622a61b28bd509aab6f"
Oct 30 07:30:18.405039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.404721    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:18.420223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.420189    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889090-d8tdg"]
Oct 30 07:30:18.432972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.432534    6738 scope.go:122] "RemoveContainer" containerID="5f38aa2ee8d6197fad1e445adce07a2c432e549f50f99614f43ec4ee324ef680"
Oct 30 07:30:18.434713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.434093    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:18.639273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.639245    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:18.642118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.642093    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:18.685797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.685765    6738 scope.go:122] "RemoveContainer" containerID="758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"
Oct 30 07:30:18.724722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.724664    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\") pod \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") "
Oct 30 07:30:18.724722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.724700    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\") pod \"a29d448b-42b0-43bb-9665-7997087515b8\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") "
Oct 30 07:30:18.724984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.724754    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\" (UniqueName: \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\") pod \"a29d448b-42b0-43bb-9665-7997087515b8\" (UID: \"a29d448b-42b0-43bb-9665-7997087515b8\") "
Oct 30 07:30:18.725068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.725042    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca" pod "0871b9f2-fc43-473b-8e0c-ebe9eb897330" (UID: "0871b9f2-fc43-473b-8e0c-ebe9eb897330"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:18.725143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.725098    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\" (UniqueName: \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\") pod \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\" (UID: \"0871b9f2-fc43-473b-8e0c-ebe9eb897330\") "
Oct 30 07:30:18.725143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.725118    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca" pod "a29d448b-42b0-43bb-9665-7997087515b8" (UID: "a29d448b-42b0-43bb-9665-7997087515b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:18.726356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.726330    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0871b9f2-fc43-473b-8e0c-ebe9eb897330-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.726705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.726558    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29d448b-42b0-43bb-9665-7997087515b8-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.728792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.728738    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn" pod "a29d448b-42b0-43bb-9665-7997087515b8" (UID: "a29d448b-42b0-43bb-9665-7997087515b8"). InnerVolumeSpecName "kube-api-access-6vfdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:18.728792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.728754    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf" pod "0871b9f2-fc43-473b-8e0c-ebe9eb897330" (UID: "0871b9f2-fc43-473b-8e0c-ebe9eb897330"). InnerVolumeSpecName "kube-api-access-9d4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:18.829191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.829100    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4cf\" (UniqueName: \"kubernetes.io/projected/0871b9f2-fc43-473b-8e0c-ebe9eb897330-kube-api-access-9d4cf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.829191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:18.829137    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfdn\" (UniqueName: \"kubernetes.io/projected/a29d448b-42b0-43bb-9665-7997087515b8-kube-api-access-6vfdn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.250637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.250603    6738 generic.go:356] "Generic (PLEG): container finished" podID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerID="a8b70f50191bcae8ad4cc81ad145e4e0b1dde5e450f60bc745bf40741b381f34" exitCode=60
Oct 30 07:30:19.251132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.250677    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerDied","Data":"a8b70f50191bcae8ad4cc81ad145e4e0b1dde5e450f60bc745bf40741b381f34"}
Oct 30 07:30:19.251132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.250744    6738 scope.go:122] "RemoveContainer" containerID="1be8afc3a23273bb8ad8ea3963e897bd7e3126b8514a39403187da20dac97a48"
Oct 30 07:30:19.251266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.251240    6738 scope.go:122] "RemoveContainer" containerID="a8b70f50191bcae8ad4cc81ad145e4e0b1dde5e450f60bc745bf40741b381f34"
Oct 30 07:30:19.251822 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:19.251720    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2_odh-ai-gateway-infra(7becc556-7eff-4b9f-9bd7-4041fd30c043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043"
Oct 30 07:30:19.252432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.252412    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"
Oct 30 07:30:19.252554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.252446    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx" event={"ID":"a29d448b-42b0-43bb-9665-7997087515b8","Type":"ContainerDied","Data":"c9c0ff54e569b8e07e68ded12307a8b2ddfd521e170aca48fb86745ebc09f3b0"}
Oct 30 07:30:19.256871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.256843    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerStarted","Data":"894529397adb2e50b69371588214d58c068a29ac39d150061036e38ca49f2ba0"}
Oct 30 07:30:19.261692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.261669    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"
Oct 30 07:30:19.261820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.261697    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf" event={"ID":"0871b9f2-fc43-473b-8e0c-ebe9eb897330","Type":"ContainerDied","Data":"ca190a8a64554db585e9101868f1ed0637d83341f920f543feb5d0593808c6c3"}
Oct 30 07:30:19.272398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.270034    6738 scope.go:122] "RemoveContainer" containerID="71bedcd5f19a217371b802f47a974610271cc8672229e5723af31d2c6fb1e4cc"
Oct 30 07:30:19.284387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.284345    6738 scope.go:122] "RemoveContainer" containerID="4e6b119cee307b83bf857294e0493f6f02b0b70cb1c91724243cc54269c13b75"
Oct 30 07:30:19.346390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.346362    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:19.356777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.356744    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889090-89jtx"]
Oct 30 07:30:19.397346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.396120    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:19.402852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.402308    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889090-cn4nf"]
Oct 30 07:30:19.476106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.473555    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:19.529641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.529612    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:19.530127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.530099    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:19.572885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.572851    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:19.659033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.658996    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\" (UniqueName: \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\") pod \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") "
Oct 30 07:30:19.659555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.659524    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\") pod \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\" (UID: \"e7e60706-5eb9-47c9-8611-a8dd8202ec7b\") "
Oct 30 07:30:19.660045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.659936    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca" pod "e7e60706-5eb9-47c9-8611-a8dd8202ec7b" (UID: "e7e60706-5eb9-47c9-8611-a8dd8202ec7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:19.660249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.660231    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.665464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.665423    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844" pod "e7e60706-5eb9-47c9-8611-a8dd8202ec7b" (UID: "e7e60706-5eb9-47c9-8611-a8dd8202ec7b"). InnerVolumeSpecName "kube-api-access-jq844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:19.698040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.698002    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" path="/var/lib/kubelet/pods/0871b9f2-fc43-473b-8e0c-ebe9eb897330/volumes"
Oct 30 07:30:19.698885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.698861    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29d448b-42b0-43bb-9665-7997087515b8" path="/var/lib/kubelet/pods/a29d448b-42b0-43bb-9665-7997087515b8/volumes"
Oct 30 07:30:19.699372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.699356    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" path="/var/lib/kubelet/pods/a64d6681-ac3f-4e97-a2f0-1483bd566653/volumes"
Oct 30 07:30:19.747935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.747908    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:19.762461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.762424    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq844\" (UniqueName: \"kubernetes.io/projected/e7e60706-5eb9-47c9-8611-a8dd8202ec7b-kube-api-access-jq844\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.863576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.863480    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\") pod \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") "
Oct 30 07:30:19.863764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.863637    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\" (UniqueName: \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\") pod \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\" (UID: \"3a8cce03-fd94-4e26-8a24-de16b92cf71d\") "
Oct 30 07:30:19.863884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.863858    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca" pod "3a8cce03-fd94-4e26-8a24-de16b92cf71d" (UID: "3a8cce03-fd94-4e26-8a24-de16b92cf71d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:19.864801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.864771    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8cce03-fd94-4e26-8a24-de16b92cf71d-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.866472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.866443    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm" pod "3a8cce03-fd94-4e26-8a24-de16b92cf71d" (UID: "3a8cce03-fd94-4e26-8a24-de16b92cf71d"). InnerVolumeSpecName "kube-api-access-k49gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:19.887249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.887223    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:19.965964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.965929    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\") pod \"230a268c-877c-4dde-9dbc-0928c029afc9\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") "
Oct 30 07:30:19.966176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.966051    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\" (UniqueName: \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\") pod \"230a268c-877c-4dde-9dbc-0928c029afc9\" (UID: \"230a268c-877c-4dde-9dbc-0928c029afc9\") "
Oct 30 07:30:19.966331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.966304    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca" pod "230a268c-877c-4dde-9dbc-0928c029afc9" (UID: "230a268c-877c-4dde-9dbc-0928c029afc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:19.966725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.966707    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k49gm\" (UniqueName: \"kubernetes.io/projected/3a8cce03-fd94-4e26-8a24-de16b92cf71d-kube-api-access-k49gm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.966808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.966729    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230a268c-877c-4dde-9dbc-0928c029afc9-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.968392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:19.968364    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9" pod "230a268c-877c-4dde-9dbc-0928c029afc9" (UID: "230a268c-877c-4dde-9dbc-0928c029afc9"). InnerVolumeSpecName "kube-api-access-mhrn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:20.068212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.068162    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrn9\" (UniqueName: \"kubernetes.io/projected/230a268c-877c-4dde-9dbc-0928c029afc9-kube-api-access-mhrn9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.144260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.144223    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:20.285550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.285500    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8" event={"ID":"230a268c-877c-4dde-9dbc-0928c029afc9","Type":"ContainerDied","Data":"2630defbe105a5e4f098e6aedb0243418301ae698e260ff5d0dbc06236771e85"}
Oct 30 07:30:20.286065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.285554    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"
Oct 30 07:30:20.286065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.285585    6738 scope.go:122] "RemoveContainer" containerID="305975dea8394a2479db441dd1688642e74ecd4d682ecd3d77893890067a45ca"
Oct 30 07:30:20.288351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.288323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerDied","Data":"894529397adb2e50b69371588214d58c068a29ac39d150061036e38ca49f2ba0"}
Oct 30 07:30:20.288497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.288330    6738 generic.go:356] "Generic (PLEG): container finished" podID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerID="894529397adb2e50b69371588214d58c068a29ac39d150061036e38ca49f2ba0" exitCode=60
Oct 30 07:30:20.288874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.288852    6738 scope.go:122] "RemoveContainer" containerID="894529397adb2e50b69371588214d58c068a29ac39d150061036e38ca49f2ba0"
Oct 30 07:30:20.289385 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:20.289350    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8_odh-ai-gateway-infra(4d91bde4-c915-4bb5-991c-5a0884d0a207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207"
Oct 30 07:30:20.290318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.290190    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"
Oct 30 07:30:20.290318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.290233    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v" event={"ID":"e7e60706-5eb9-47c9-8611-a8dd8202ec7b","Type":"ContainerDied","Data":"a5220eafd520251f2e9c4b5f85b9fffafd13f4d8a9e535448b1bd000656264d7"}
Oct 30 07:30:20.293165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.293149    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"
Oct 30 07:30:20.293247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.293215    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9" event={"ID":"3a8cce03-fd94-4e26-8a24-de16b92cf71d","Type":"ContainerDied","Data":"16c0cc02c0e764b45025f18927589b7b6547137abdcfa5f86dc857b5b65936ac"}
Oct 30 07:30:20.306728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.306705    6738 scope.go:122] "RemoveContainer" containerID="758b1112a56e7635fcbc1df6050066e11bab0008d44ea2eee6117cc45e253af6"
Oct 30 07:30:20.323500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.323478    6738 scope.go:122] "RemoveContainer" containerID="40832e52e8185c886ac6d07d28d4b9d2ed1355f4e1ec5ad7df3822c8c201f2a5"
Oct 30 07:30:20.335805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.335781    6738 scope.go:122] "RemoveContainer" containerID="abd04214842f89e9bcb25625c5fd1ad509ec235c7d7b00da2dfc95265cccfdfa"
Oct 30 07:30:20.349584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.345458    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:20.351871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.350997    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889090-tbl5v"]
Oct 30 07:30:20.486131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.486089    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:20.492240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.492209    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:20.492623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.492607    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:20.497993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.497958    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889090-n7dz8"]
Oct 30 07:30:20.555956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.555915    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:20.566269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.566201    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889090-7bsp9"]
Oct 30 07:30:20.591225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.591190    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\" (UniqueName: \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\") pod \"7becc556-7eff-4b9f-9bd7-4041fd30c043\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") "
Oct 30 07:30:20.591432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.591250    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\") pod \"7becc556-7eff-4b9f-9bd7-4041fd30c043\" (UID: \"7becc556-7eff-4b9f-9bd7-4041fd30c043\") "
Oct 30 07:30:20.591668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.591645    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca" pod "7becc556-7eff-4b9f-9bd7-4041fd30c043" (UID: "7becc556-7eff-4b9f-9bd7-4041fd30c043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:20.592001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.591984    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7becc556-7eff-4b9f-9bd7-4041fd30c043-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.593627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.593598    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc" pod "7becc556-7eff-4b9f-9bd7-4041fd30c043" (UID: "7becc556-7eff-4b9f-9bd7-4041fd30c043"). InnerVolumeSpecName "kube-api-access-4kwhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:20.694002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:20.693964    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwhc\" (UniqueName: \"kubernetes.io/projected/7becc556-7eff-4b9f-9bd7-4041fd30c043-kube-api-access-4kwhc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:21.320226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.320197    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"
Oct 30 07:30:21.320721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.320216    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2" event={"ID":"7becc556-7eff-4b9f-9bd7-4041fd30c043","Type":"ContainerDied","Data":"03c14fc7e34d4233377e3cf76ed44eda2a551fed21f11baa3874dda19fdffb0b"}
Oct 30 07:30:21.320721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.320362    6738 scope.go:122] "RemoveContainer" containerID="a8b70f50191bcae8ad4cc81ad145e4e0b1dde5e450f60bc745bf40741b381f34"
Oct 30 07:30:21.375347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.375313    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:21.387934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.387908    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889090-fksx2"]
Oct 30 07:30:21.510609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.510583    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:21.615759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.615679    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\") pod \"4d91bde4-c915-4bb5-991c-5a0884d0a207\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") "
Oct 30 07:30:21.615978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.615952    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\" (UniqueName: \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\") pod \"4d91bde4-c915-4bb5-991c-5a0884d0a207\" (UID: \"4d91bde4-c915-4bb5-991c-5a0884d0a207\") "
Oct 30 07:30:21.616287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.616265    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca" pod "4d91bde4-c915-4bb5-991c-5a0884d0a207" (UID: "4d91bde4-c915-4bb5-991c-5a0884d0a207"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:21.616637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.616619    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d91bde4-c915-4bb5-991c-5a0884d0a207-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:21.618410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.618382    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9" pod "4d91bde4-c915-4bb5-991c-5a0884d0a207" (UID: "4d91bde4-c915-4bb5-991c-5a0884d0a207"). InnerVolumeSpecName "kube-api-access-hskz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:21.691586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.691549    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" path="/var/lib/kubelet/pods/230a268c-877c-4dde-9dbc-0928c029afc9/volumes"
Oct 30 07:30:21.692206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.692174    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" path="/var/lib/kubelet/pods/3a8cce03-fd94-4e26-8a24-de16b92cf71d/volumes"
Oct 30 07:30:21.692742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.692713    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" path="/var/lib/kubelet/pods/7becc556-7eff-4b9f-9bd7-4041fd30c043/volumes"
Oct 30 07:30:21.693580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.693560    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" path="/var/lib/kubelet/pods/e7e60706-5eb9-47c9-8611-a8dd8202ec7b/volumes"
Oct 30 07:30:21.718013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:21.717976    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hskz9\" (UniqueName: \"kubernetes.io/projected/4d91bde4-c915-4bb5-991c-5a0884d0a207-kube-api-access-hskz9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:22.065932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.065893    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:22.335614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.335533    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"
Oct 30 07:30:22.335614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.335575    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8" event={"ID":"4d91bde4-c915-4bb5-991c-5a0884d0a207","Type":"ContainerDied","Data":"c88c6736c62ca244c78ea4c28567c8ee228239cffda2a892ebb8a370f96dc1a5"}
Oct 30 07:30:22.336133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.335644    6738 scope.go:122] "RemoveContainer" containerID="894529397adb2e50b69371588214d58c068a29ac39d150061036e38ca49f2ba0"
Oct 30 07:30:22.392237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.392203    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:22.398561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:22.398529    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889090-lflf8"]
Oct 30 07:30:23.697615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:23.697573    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" path="/var/lib/kubelet/pods/4d91bde4-c915-4bb5-991c-5a0884d0a207/volumes"
Oct 30 07:30:24.034579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:24.034498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:25.033183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:25.033150    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"]
Oct 30 07:30:25.042987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:25.042951    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e3423-klr6x"]
Oct 30 07:30:25.691768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:25.691733    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b711cbd8-08b9-4b77-a145-adc6e8349311" path="/var/lib/kubelet/pods/b711cbd8-08b9-4b77-a145-adc6e8349311/volumes"
Oct 30 07:30:25.933720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:25.933682    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:28.193065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.193035    6738 scope.go:122] "RemoveContainer" containerID="0bd5644d83e5aaaf6283c3418ab91ef594eb1b367807f7b05e3ca74d9bb3e3e6"
Oct 30 07:30:28.207192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.207171    6738 scope.go:122] "RemoveContainer" containerID="599b1822215d1cddbc26daaac08351d363f48d485aa71fce757c415805238229"
Oct 30 07:30:28.220821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.220797    6738 scope.go:122] "RemoveContainer" containerID="79a998056639e24432acee15dbb9a4fb3f7946c361794b1a51b75295620b0804"
Oct 30 07:30:28.239152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.239125    6738 scope.go:122] "RemoveContainer" containerID="96832e11b0f564d78e93578866acc6643eee8638b3d2f5b8a2f4ff5b558e9e17"
Oct 30 07:30:28.239284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.239258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:28.250036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.250011    6738 scope.go:122] "RemoveContainer" containerID="3c9753a433264b3468898acbaa257f10b540a48652e9438419920452ca59d704"
Oct 30 07:30:28.259992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.259971    6738 scope.go:122] "RemoveContainer" containerID="6cce4611c0aae78ba5a2c93763030d6edd881c66d062a1a4b5487e5704c1186a"
Oct 30 07:30:28.273180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.273150    6738 scope.go:122] "RemoveContainer" containerID="bcd255555ae94d0c2272bff352a441ae9b983382a2c9e7197157ee3f822754d7"
Oct 30 07:30:28.384635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.381395    6738 scope.go:122] "RemoveContainer" containerID="f88003af1cc1fbf18048ebe2bb699ddb2ee59900197d4a9a2a6e7a1ee9ebf31d"
Oct 30 07:30:28.432074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.430910    6738 scope.go:122] "RemoveContainer" containerID="8d4d898473080dc7800bf1940213c15dab50a21e7f67860b6c120f77d2805134"
Oct 30 07:30:28.443775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:28.443752    6738 scope.go:122] "RemoveContainer" containerID="3292c7370319f719a3a463519f7c2fe1495229a4e25814c8f002ca91b9be891f"
Oct 30 07:30:30.219237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:30.219136    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:31.591517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:31.591483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:32.131837 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:32.131797    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:33.880002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:33.879961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:35.722843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:35.722807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:37.521544 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:37.521503    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:30:37.598938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:37.598907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:39.565739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:39.562756    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:41.253006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:41.252970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:43.338594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:43.338551    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:45.322622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:45.322568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:47.084668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.084622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:47.221162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.221127    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:47.373555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.373517    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:30:47.374070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.374033    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy" containerID="cri-o://4f31bfb861fed986959a2f28426f599fc9d219fddb466668ecd9f0df022614ac" gracePeriod=30
Oct 30 07:30:47.386965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.386933    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:30:47.387231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.387203    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy" containerID="cri-o://ce35f28a00f1193fd282d942a5aa4ae1256c6b81c63cc2cf168b05eebba71898" gracePeriod=30
Oct 30 07:30:47.392265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.392236    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:30:47.392515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.392491    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy" containerID="cri-o://accb17c965516b2a7b73d9c94dd46f78333ae53e667a00960b3f185792473d2d" gracePeriod=30
Oct 30 07:30:47.398649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.398622    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:30:47.398820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.398802    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy" containerID="cri-o://5f0410dff540774ec3fdbe8cc8570fbe31822f831e1101da80bac2b3f1074532" gracePeriod=30
Oct 30 07:30:47.638526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:47.638399    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:48.179953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:48.179916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:48.676465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:48.676430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:49.088183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:49.088090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:50.966620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:50.966581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:52.490983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.490953    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.688579    6738 generic.go:356] "Generic (PLEG): container finished" podID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerID="5f0410dff540774ec3fdbe8cc8570fbe31822f831e1101da80bac2b3f1074532" exitCode=0
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.688625    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" event={"ID":"a812e3c0-93d1-45ad-9df8-6467858de0f1","Type":"ContainerDied","Data":"5f0410dff540774ec3fdbe8cc8570fbe31822f831e1101da80bac2b3f1074532"}
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.690884    6738 generic.go:356] "Generic (PLEG): container finished" podID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerID="4f31bfb861fed986959a2f28426f599fc9d219fddb466668ecd9f0df022614ac" exitCode=0
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.691155    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" event={"ID":"f7a0f1ce-51ab-4fec-9879-5717948d191f","Type":"ContainerDied","Data":"4f31bfb861fed986959a2f28426f599fc9d219fddb466668ecd9f0df022614ac"}
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.693283    6738 generic.go:356] "Generic (PLEG): container finished" podID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerID="ce35f28a00f1193fd282d942a5aa4ae1256c6b81c63cc2cf168b05eebba71898" exitCode=0
Oct 30 07:30:52.694536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.693391    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" event={"ID":"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b","Type":"ContainerDied","Data":"ce35f28a00f1193fd282d942a5aa4ae1256c6b81c63cc2cf168b05eebba71898"}
Oct 30 07:30:52.695357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.695328    6738 generic.go:356] "Generic (PLEG): container finished" podID="a23e079a-adb3-4c16-9450-82ac846243b9" containerID="accb17c965516b2a7b73d9c94dd46f78333ae53e667a00960b3f185792473d2d" exitCode=0
Oct 30 07:30:52.695509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.695383    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" event={"ID":"a23e079a-adb3-4c16-9450-82ac846243b9","Type":"ContainerDied","Data":"accb17c965516b2a7b73d9c94dd46f78333ae53e667a00960b3f185792473d2d"}
Oct 30 07:30:52.732415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.732381    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:30:52.836860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.836832    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.837087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837067    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.837311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837283    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.837518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837503    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.837673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837657    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.837868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837851    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.838013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.838000    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.838170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.838156    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.838286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.838274    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\") pod \"f7a0f1ce-51ab-4fec-9879-5717948d191f\" (UID: \"f7a0f1ce-51ab-4fec-9879-5717948d191f\") "
Oct 30 07:30:52.839091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.837686    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.839091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.838076    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.841712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.839411    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.841712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.839439    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a0f1ce-51ab-4fec-9879-5717948d191f-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.841712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.839464    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.841712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.839603    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.841712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.841092    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.847215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.846855    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.847555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.847518    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.848413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.848376    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "kube-api-access-dqgbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.849899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.849875    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:52.850237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.850166    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo" pod "f7a0f1ce-51ab-4fec-9879-5717948d191f" (UID: "f7a0f1ce-51ab-4fec-9879-5717948d191f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.856064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.855723    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:30:52.858721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.858697    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:30:52.865063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.865032    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:30:52.940835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.940801    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.941046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.940855    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.940908    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.940940    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.940983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.941046 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941041    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941139    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941157    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.941375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941188    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.941375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941238    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.941375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941272    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.941972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.941778    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.942081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.942081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942030    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.942205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942073    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.942272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942224    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.942272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942260    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942420    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942471    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942511    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942541    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.942607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942556    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.942951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942876    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.943017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942951    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.943017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.942985    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.943129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943088    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.943185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943133    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.943185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943165    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.943307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943228    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\") pod \"a812e3c0-93d1-45ad-9df8-6467858de0f1\" (UID: \"a812e3c0-93d1-45ad-9df8-6467858de0f1\") "
Oct 30 07:30:52.943307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943256    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.943426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943330    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\") pod \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\" (UID: \"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b\") "
Oct 30 07:30:52.943426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.943369    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\") pod \"a23e079a-adb3-4c16-9450-82ac846243b9\" (UID: \"a23e079a-adb3-4c16-9450-82ac846243b9\") "
Oct 30 07:30:52.944440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.944418    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.944623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.944576    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945439    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945464    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945480    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945494    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945508    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945521    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a812e3c0-93d1-45ad-9df8-6467858de0f1-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945537    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945551    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945565    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945577    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a0f1ce-51ab-4fec-9879-5717948d191f-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945591    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgbc\" (UniqueName: \"kubernetes.io/projected/f7a0f1ce-51ab-4fec-9879-5717948d191f-kube-api-access-dqgbc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.944722    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.944948    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.944982    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945155    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945407    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.946470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.945839    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.946470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.946122    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.946595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.946479    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.946746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.946717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.947634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.947611    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "kube-api-access-qgl2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.950982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.950953    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.951788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.951762    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.951997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.951813    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "kube-api-access-scbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.952081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.952081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952026    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "kube-api-access-cdqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.952244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952201    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.952409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952280    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.952409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952303    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo" pod "a812e3c0-93d1-45ad-9df8-6467858de0f1" (UID: "a812e3c0-93d1-45ad-9df8-6467858de0f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.952409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952318    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.952409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.952360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy" pod "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" (UID: "b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.956372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:52.956289    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo" pod "a23e079a-adb3-4c16-9450-82ac846243b9" (UID: "a23e079a-adb3-4c16-9450-82ac846243b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:53.047255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047215    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047256    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047269    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047285    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047316    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047329    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047342    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23e079a-adb3-4c16-9450-82ac846243b9-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047356    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047370    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdqpf\" (UniqueName: \"kubernetes.io/projected/a812e3c0-93d1-45ad-9df8-6467858de0f1-kube-api-access-cdqpf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047383    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047396    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047410    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047422    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23e079a-adb3-4c16-9450-82ac846243b9-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047434    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbqd\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-kube-api-access-scbqd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047445    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047457    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047469    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23e079a-adb3-4c16-9450-82ac846243b9-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047485    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a812e3c0-93d1-45ad-9df8-6467858de0f1-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.047497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047498    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgl2r\" (UniqueName: \"kubernetes.io/projected/a23e079a-adb3-4c16-9450-82ac846243b9-kube-api-access-qgl2r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.048050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047510    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.048050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.047525    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:53.388501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.388470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:53.537614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.537549    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:30:53.538124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.537777    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy" containerID="cri-o://26911ed328b001486e80932fd8b79e9f445f6b05efcbf638c3fc76b47630e362" gracePeriod=30
Oct 30 07:30:53.544404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.544232    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:30:53.544569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.544502    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy" containerID="cri-o://f5aea6022f24fa0a884778ef2fcc5efdce975024f4de9182a55c09464aa1ee15" gracePeriod=30
Oct 30 07:30:53.552679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.552650    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:30:53.552981 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.552945    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy" containerID="cri-o://0cad1a6fe6f701991a29ea38edf7d8486a199bbed4ae01c521d5cfdd8025ea03" gracePeriod=30
Oct 30 07:30:53.557565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.557537    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:30:53.557976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.557946    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy" containerID="cri-o://91bc7948c650440d329e6dd41c0160744a67559898937337e7d736c19d3eb900" gracePeriod=30
Oct 30 07:30:53.709717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.706061    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"
Oct 30 07:30:53.709717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.706110    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p" event={"ID":"f7a0f1ce-51ab-4fec-9879-5717948d191f","Type":"ContainerDied","Data":"fcfd3292aa0865ffbafaf01ad6ea69454002599152537734c0a69c0ba0760a08"}
Oct 30 07:30:53.709717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.706175    6738 scope.go:122] "RemoveContainer" containerID="4f31bfb861fed986959a2f28426f599fc9d219fddb466668ecd9f0df022614ac"
Oct 30 07:30:53.709717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.708249    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"
Oct 30 07:30:53.709717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.708269    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn" event={"ID":"b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b","Type":"ContainerDied","Data":"025b1d55760b39078d2a8df60b9ab45cdc684fbbfecfe3d53183a50200f6f39a"}
Oct 30 07:30:53.710287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.710193    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59" event={"ID":"a23e079a-adb3-4c16-9450-82ac846243b9","Type":"ContainerDied","Data":"76952dd0262f5df0f45eeca79e7908ab2e84c956cb10dc6a316f4c47b7bc189d"}
Oct 30 07:30:53.710287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.710259    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"
Oct 30 07:30:53.712823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.712709    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7" event={"ID":"a812e3c0-93d1-45ad-9df8-6467858de0f1","Type":"ContainerDied","Data":"0e94b660b720368e612d0fe12b56aea98a071f4fe256c37c23377e0a2565062c"}
Oct 30 07:30:53.712823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.712797    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"
Oct 30 07:30:53.722940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.722914    6738 scope.go:122] "RemoveContainer" containerID="ce35f28a00f1193fd282d942a5aa4ae1256c6b81c63cc2cf168b05eebba71898"
Oct 30 07:30:53.734423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.734395    6738 scope.go:122] "RemoveContainer" containerID="accb17c965516b2a7b73d9c94dd46f78333ae53e667a00960b3f185792473d2d"
Oct 30 07:30:53.745257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.745234    6738 scope.go:122] "RemoveContainer" containerID="5f0410dff540774ec3fdbe8cc8570fbe31822f831e1101da80bac2b3f1074532"
Oct 30 07:30:53.774285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.774232    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:30:53.777907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.777883    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-xbh59"]
Oct 30 07:30:53.820443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.820364    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:30:53.825338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.825255    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-4gtmn"]
Oct 30 07:30:53.851826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.851788    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:30:53.857362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.857333    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-9sgt7"]
Oct 30 07:30:53.891621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.891556    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:30:53.895356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:53.895328    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-jfz8p"]
Oct 30 07:30:54.163889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.163852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:54.275051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.275016    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:54.335355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.335314    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 07:30:54.336093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336074    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336094    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336111    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336117    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336124    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336129    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336140    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336145    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336152    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336157    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336167    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336172    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336182    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336186    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336194    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336199    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336209    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336213    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336220    6738 cpu_manager.go:414] "containerMap: removing container" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336225    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336230    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336235    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336241    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336245    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336251    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336255    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336263    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336267    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336277    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336282    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336288    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336306    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336319    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336324    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336329    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336334    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336342    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336347    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336358    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336363    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336370    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336374    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336382    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336387    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336398    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336403    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336408    6738 cpu_manager.go:414] "containerMap: removing container" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336413    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336424    6738 cpu_manager.go:414] "containerMap: removing container" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.336460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336429    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336553    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336561    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336570    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336575    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336581    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336587    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e60706-5eb9-47c9-8611-a8dd8202ec7b" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336595    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" containerName="istio-proxy"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336600    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336605    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336612    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336618    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336623    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336628    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336633    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336640    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8cce03-fd94-4e26-8a24-de16b92cf71d" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336647    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336653    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336660    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336666    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336672    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" containerName="istio-proxy"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336677    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" containerName="istio-proxy"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336683    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d448b-42b0-43bb-9665-7997087515b8" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336689    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336696    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" containerName="istio-proxy"
Oct 30 07:30:54.337727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.336702    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.342305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.342268    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.357573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.357543    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d09eb6-dockercfg-6b5km\""
Oct 30 07:30:54.357573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.357544    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d09eb6\""
Oct 30 07:30:54.358538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.358511    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 07:30:54.371700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.371661    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkq5\" (UniqueName: \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.371895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.371870    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.398784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.398741    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 07:30:54.402313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.400169    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.402565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.402540    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7becc556-7eff-4b9f-9bd7-4041fd30c043" containerName="cleanup"
Oct 30 07:30:54.402774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.402760    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.402909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.402894    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0871b9f2-fc43-473b-8e0c-ebe9eb897330" containerName="cleanup"
Oct 30 07:30:54.403055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.403042    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.403147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.403136    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.403509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.403490    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64d6681-ac3f-4e97-a2f0-1483bd566653" containerName="cleanup"
Oct 30 07:30:54.403663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.403651    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="230a268c-877c-4dde-9dbc-0928c029afc9" containerName="cleanup"
Oct 30 07:30:54.403778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.403762    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d91bde4-c915-4bb5-991c-5a0884d0a207" containerName="cleanup"
Oct 30 07:30:54.432368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.430612    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.602213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.573537    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.602213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.573778    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.602213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.573836    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4z6\" (UniqueName: \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.602213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.573899    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkq5\" (UniqueName: \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.602213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.574987    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.617438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.617403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkq5\" (UniqueName: \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\") pod \"payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.651172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.650355    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 07:30:54.670319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.660584    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:54.692374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.684076    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.692374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.684210    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4z6\" (UniqueName: \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.692374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.685561    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.770113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.770082    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4z6\" (UniqueName: \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\") pod \"payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:54.931922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:54.931884    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:55.067098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.067062    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 07:30:55.067249 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:55.067121    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfaa571_f6ba_45da_bf89_b8448fa096b2.slice/crio-98c68dedd2ec6d89701aa1c0e7f22017a81463b1b0a79ae08ca00e028d978959 WatchSource:0}: Error finding container 98c68dedd2ec6d89701aa1c0e7f22017a81463b1b0a79ae08ca00e028d978959: Status 404 returned error can't find the container with id 98c68dedd2ec6d89701aa1c0e7f22017a81463b1b0a79ae08ca00e028d978959
Oct 30 07:30:55.247747 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:30:55.247712    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53090933_8414_4e71_aa9f_b79991927dc7.slice/crio-12fcf4514be22f61d832d315b7a4fd3e0a75303624bbb5c51c8faa98a3e1cb2e WatchSource:0}: Error finding container 12fcf4514be22f61d832d315b7a4fd3e0a75303624bbb5c51c8faa98a3e1cb2e: Status 404 returned error can't find the container with id 12fcf4514be22f61d832d315b7a4fd3e0a75303624bbb5c51c8faa98a3e1cb2e
Oct 30 07:30:55.248111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.248086    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 07:30:55.449787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.449750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:55.694100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.694058    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23e079a-adb3-4c16-9450-82ac846243b9" path="/var/lib/kubelet/pods/a23e079a-adb3-4c16-9450-82ac846243b9/volumes"
Oct 30 07:30:55.694912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.694887    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a812e3c0-93d1-45ad-9df8-6467858de0f1" path="/var/lib/kubelet/pods/a812e3c0-93d1-45ad-9df8-6467858de0f1/volumes"
Oct 30 07:30:55.695573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.695549    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b" path="/var/lib/kubelet/pods/b8e1ad14-0cbb-4be0-8eaf-0518ffe7596b/volumes"
Oct 30 07:30:55.696680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.696657    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a0f1ce-51ab-4fec-9879-5717948d191f" path="/var/lib/kubelet/pods/f7a0f1ce-51ab-4fec-9879-5717948d191f/volumes"
Oct 30 07:30:55.794533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.794479    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" event={"ID":"3cfaa571-f6ba-45da-bf89-b8448fa096b2","Type":"ContainerStarted","Data":"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"}
Oct 30 07:30:55.794744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.794553    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" event={"ID":"3cfaa571-f6ba-45da-bf89-b8448fa096b2","Type":"ContainerStarted","Data":"98c68dedd2ec6d89701aa1c0e7f22017a81463b1b0a79ae08ca00e028d978959"}
Oct 30 07:30:55.794744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.794674    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:30:55.796592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.796552    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" event={"ID":"53090933-8414-4e71-aa9f-b79991927dc7","Type":"ContainerStarted","Data":"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"}
Oct 30 07:30:55.796738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.796603    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" event={"ID":"53090933-8414-4e71-aa9f-b79991927dc7","Type":"ContainerStarted","Data":"12fcf4514be22f61d832d315b7a4fd3e0a75303624bbb5c51c8faa98a3e1cb2e"}
Oct 30 07:30:55.796738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.796655    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:30:55.837557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.837500    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" podStartSLOduration=1.8374849439999998 podStartE2EDuration="1.837484944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:55.835547257 +0000 UTC m=+2016.268311151" watchObservedRunningTime="2026-10-30 07:30:55.837484944 +0000 UTC m=+2016.270248836"
Oct 30 07:30:55.879000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:55.878937    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" podStartSLOduration=1.8789232679999999 podStartE2EDuration="1.878923268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:55.876084251 +0000 UTC m=+2016.308848144" watchObservedRunningTime="2026-10-30 07:30:55.878923268 +0000 UTC m=+2016.311687160"
Oct 30 07:30:57.355017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:57.354985    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:58.822455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.822373    6738 generic.go:356] "Generic (PLEG): container finished" podID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerID="91bc7948c650440d329e6dd41c0160744a67559898937337e7d736c19d3eb900" exitCode=0
Oct 30 07:30:58.822455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.822430    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" event={"ID":"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0","Type":"ContainerDied","Data":"91bc7948c650440d329e6dd41c0160744a67559898937337e7d736c19d3eb900"}
Oct 30 07:30:58.824528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.824502    6738 generic.go:356] "Generic (PLEG): container finished" podID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerID="0cad1a6fe6f701991a29ea38edf7d8486a199bbed4ae01c521d5cfdd8025ea03" exitCode=0
Oct 30 07:30:58.824637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.824573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" event={"ID":"a5c7b524-2ec7-46f2-ae32-c33291319f81","Type":"ContainerDied","Data":"0cad1a6fe6f701991a29ea38edf7d8486a199bbed4ae01c521d5cfdd8025ea03"}
Oct 30 07:30:58.826544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.826516    6738 generic.go:356] "Generic (PLEG): container finished" podID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerID="f5aea6022f24fa0a884778ef2fcc5efdce975024f4de9182a55c09464aa1ee15" exitCode=0
Oct 30 07:30:58.826719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.826601    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" event={"ID":"6de7d078-4ca1-4141-a97e-030055b71ca1","Type":"ContainerDied","Data":"f5aea6022f24fa0a884778ef2fcc5efdce975024f4de9182a55c09464aa1ee15"}
Oct 30 07:30:58.828814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.828792    6738 generic.go:356] "Generic (PLEG): container finished" podID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerID="26911ed328b001486e80932fd8b79e9f445f6b05efcbf638c3fc76b47630e362" exitCode=0
Oct 30 07:30:58.828964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.828881    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" event={"ID":"3ea59724-25db-44f8-b816-a6e85bf97b7c","Type":"ContainerDied","Data":"26911ed328b001486e80932fd8b79e9f445f6b05efcbf638c3fc76b47630e362"}
Oct 30 07:30:58.852347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.852267    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:30:58.921407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.921374    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:30:58.928128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:58.928098    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:30:59.003584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003548    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.003805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003615    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.003805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003643    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.003805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003692    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.003805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003725    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.003805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003752    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003809    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003844    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003903    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.003985    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004006    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004019    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004052    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.004147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004097    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.004638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004174    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.004638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004215    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.004638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004248    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.004638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004303    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.004638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004530    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.004925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004687    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.004925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004701    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.004925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.004736    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.005093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005013    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:59.005147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005097    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:59.005247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005227    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.005358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005333    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.005435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005372    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.005435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005411    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.006044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005636    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.006044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005683    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.006214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.005718    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.006214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006025    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.006214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006135    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.006408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006241    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.006408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006285    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.006408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006368    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.006569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006422    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\") pod \"6de7d078-4ca1-4141-a97e-030055b71ca1\" (UID: \"6de7d078-4ca1-4141-a97e-030055b71ca1\") "
Oct 30 07:30:59.006569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006475    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\") pod \"a5c7b524-2ec7-46f2-ae32-c33291319f81\" (UID: \"a5c7b524-2ec7-46f2-ae32-c33291319f81\") "
Oct 30 07:30:59.006680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.006571    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\") pod \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\" (UID: \"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0\") "
Oct 30 07:30:59.007744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.007178    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.007744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.007305    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.007744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.007390    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:59.007744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.007710    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.008926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.008903    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.008933    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.008949    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.008979    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7b524-2ec7-46f2-ae32-c33291319f81-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.008995    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009009    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009023    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009053    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009066    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009079    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009092    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009106    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009136    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009151    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009165    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de7d078-4ca1-4141-a97e-030055b71ca1-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.009409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.009181    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.010632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.010597    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "kube-api-access-94d7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.010722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.010687    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:59.010978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.010932    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.018903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.013271    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "kube-api-access-qrkxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.024631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.024551    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:59.024631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.024562    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy" pod "6de7d078-4ca1-4141-a97e-030055b71ca1" (UID: "6de7d078-4ca1-4141-a97e-030055b71ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.024631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.024592    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.024905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.024732    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.025062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.025038    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx" pod "a5c7b524-2ec7-46f2-ae32-c33291319f81" (UID: "a5c7b524-2ec7-46f2-ae32-c33291319f81"). InnerVolumeSpecName "kube-api-access-g2vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.025223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.025204    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:59.027075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.027046    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token" pod "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" (UID: "1aaf1a25-d92c-4277-9fd6-9b80bccd95b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.110988    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111019    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111028    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2vrx\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-kube-api-access-g2vrx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111038    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111048    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111058    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111066    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7b524-2ec7-46f2-ae32-c33291319f81-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111073    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111082    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrkxw\" (UniqueName: \"kubernetes.io/projected/6de7d078-4ca1-4141-a97e-030055b71ca1-kube-api-access-qrkxw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111090    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de7d078-4ca1-4141-a97e-030055b71ca1-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.111557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.111099    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94d7g\" (UniqueName: \"kubernetes.io/projected/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0-kube-api-access-94d7g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.207781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.207751    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:30:59.247682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.247643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:30:59.331319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.327542    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.338531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.338497    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343116    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343496    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343563    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343633    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "kube-api-access-sm9l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.343779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343823    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343902    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343936    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\") pod \"3ea59724-25db-44f8-b816-a6e85bf97b7c\" (UID: \"3ea59724-25db-44f8-b816-a6e85bf97b7c\") "
Oct 30 07:30:59.343982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.343939    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.344789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.344763    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9l6\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-kube-api-access-sm9l6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.344959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.344941    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.348223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.344785    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:59.348223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.345011    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.348546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.348501    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.351068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.351040    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.384581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.384371    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:59.386239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.386204    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:59.386893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.386853    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo" pod "3ea59724-25db-44f8-b816-a6e85bf97b7c" (UID: "3ea59724-25db-44f8-b816-a6e85bf97b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:59.446417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446368    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446415    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446428    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446438    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446447    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446456    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea59724-25db-44f8-b816-a6e85bf97b7c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.446765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.446464    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea59724-25db-44f8-b816-a6e85bf97b7c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:30:59.761519 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:30:59.761474    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea59724_25db_44f8_b816_a6e85bf97b7c.slice/crio-ca8f4544488871daa170ef8f78f070ae3c38f4354918eb123365aa0c48a71162\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7b524_2ec7_46f2_ae32_c33291319f81.slice/crio-ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7b524_2ec7_46f2_ae32_c33291319f81.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de7d078_4ca1_4141_a97e_030055b71ca1.slice/crio-7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaf1a25_d92c_4277_9fd6_9b80bccd95b0.slice/crio-7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a\": RecentStats: unable to find data in memory cache]"
Oct 30 07:30:59.838693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.838664    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"
Oct 30 07:30:59.839165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.838668    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk" event={"ID":"1aaf1a25-d92c-4277-9fd6-9b80bccd95b0","Type":"ContainerDied","Data":"7af48b03a16abe14085615fcef6783597d28ab3182ff33671de3d38f66af1e4a"}
Oct 30 07:30:59.839165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.838809    6738 scope.go:122] "RemoveContainer" containerID="91bc7948c650440d329e6dd41c0160744a67559898937337e7d736c19d3eb900"
Oct 30 07:30:59.840596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.840571    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"
Oct 30 07:30:59.840682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.840578    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6" event={"ID":"a5c7b524-2ec7-46f2-ae32-c33291319f81","Type":"ContainerDied","Data":"ee0dc482bada1f8b918d4420f6dca6e8dbf36fcb719f19d1eb4dda3ad17bbbd2"}
Oct 30 07:30:59.842619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.842595    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg" event={"ID":"6de7d078-4ca1-4141-a97e-030055b71ca1","Type":"ContainerDied","Data":"7ce7cb6a4bdb5b2036ab2c1a998c7b38e6bf73875df6ab0a1510ca534c227360"}
Oct 30 07:30:59.842832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.842609    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"
Oct 30 07:30:59.844680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.844654    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf" event={"ID":"3ea59724-25db-44f8-b816-a6e85bf97b7c","Type":"ContainerDied","Data":"ca8f4544488871daa170ef8f78f070ae3c38f4354918eb123365aa0c48a71162"}
Oct 30 07:30:59.844803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.844757    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"
Oct 30 07:30:59.854021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.853995    6738 scope.go:122] "RemoveContainer" containerID="0cad1a6fe6f701991a29ea38edf7d8486a199bbed4ae01c521d5cfdd8025ea03"
Oct 30 07:30:59.866662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.866637    6738 scope.go:122] "RemoveContainer" containerID="f5aea6022f24fa0a884778ef2fcc5efdce975024f4de9182a55c09464aa1ee15"
Oct 30 07:30:59.878055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.878028    6738 scope.go:122] "RemoveContainer" containerID="26911ed328b001486e80932fd8b79e9f445f6b05efcbf638c3fc76b47630e362"
Oct 30 07:30:59.905882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.905848    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:30:59.918419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.918387    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-p7fp6"]
Oct 30 07:30:59.945451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.945411    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:30:59.953962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.953864    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-t5dgk"]
Oct 30 07:30:59.985441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.985393    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:30:59.987848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:30:59.987823    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-r47nf"]
Oct 30 07:31:00.025617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:00.025563    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:31:00.030487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:00.030451    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-vqbdg"]
Oct 30 07:31:01.026514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:01.026476    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:01.691254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:01.691220    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" path="/var/lib/kubelet/pods/1aaf1a25-d92c-4277-9fd6-9b80bccd95b0/volumes"
Oct 30 07:31:01.691840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:01.691822    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" path="/var/lib/kubelet/pods/3ea59724-25db-44f8-b816-a6e85bf97b7c/volumes"
Oct 30 07:31:01.692272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:01.692258    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" path="/var/lib/kubelet/pods/6de7d078-4ca1-4141-a97e-030055b71ca1/volumes"
Oct 30 07:31:01.693070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:01.693054    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" path="/var/lib/kubelet/pods/a5c7b524-2ec7-46f2-ae32-c33291319f81/volumes"
Oct 30 07:31:02.918356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:02.918289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:04.767541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:04.767505    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:06.706795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:06.706755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:06.816916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:06.816880    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 07:31:06.818970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:06.818944    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 07:31:07.521547 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:07.521503    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:31:08.338473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.338432    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"]
Oct 30 07:31:08.339188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339170    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339189    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339200    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339205    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339237    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339242    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy"
Oct 30 07:31:08.339265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339261    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy"
Oct 30 07:31:08.339513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339267    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy"
Oct 30 07:31:08.339513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339469    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de7d078-4ca1-4141-a97e-030055b71ca1" containerName="istio-proxy"
Oct 30 07:31:08.339513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339484    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaf1a25-d92c-4277-9fd6-9b80bccd95b0" containerName="istio-proxy"
Oct 30 07:31:08.339513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339497    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c7b524-2ec7-46f2-ae32-c33291319f81" containerName="istio-proxy"
Oct 30 07:31:08.339513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.339505    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea59724-25db-44f8-b816-a6e85bf97b7c" containerName="istio-proxy"
Oct 30 07:31:08.344866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.344846    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.360332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.360282    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"]
Oct 30 07:31:08.368231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.368201    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:31:08.368231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.368196    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:31:08.408680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.408633    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.408985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.408949    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqx4w\" (UniqueName: \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.409111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.409087    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.510779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.510744    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqx4w\" (UniqueName: \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.510991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.510821    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.510991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.510891    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.511560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.511532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.513365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.513334    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.534727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.534699    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqx4w\" (UniqueName: \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\") pod \"maas-api-revoke-sub-2fae8551-6hrx2\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:08.638980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.638939    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:08.697897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:08.697863    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:09.057249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:09.057215    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"]
Oct 30 07:31:09.057687 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:09.057657    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b83d00_e8c7_4a35_b4f9_3c21a9c71257.slice/crio-4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb WatchSource:0}: Error finding container 4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb: Status 404 returned error can't find the container with id 4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb
Oct 30 07:31:09.951812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:09.951752    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2" event={"ID":"61b83d00-e8c7-4a35-b4f9-3c21a9c71257","Type":"ContainerStarted","Data":"8b299d060d33573221b23ad0bedc7f4b7f24c8f27746a4bbc9c861e968e591e4"}
Oct 30 07:31:09.952320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:09.951835    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2" event={"ID":"61b83d00-e8c7-4a35-b4f9-3c21a9c71257","Type":"ContainerStarted","Data":"4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb"}
Oct 30 07:31:09.994325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:09.994229    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2" podStartSLOduration=1.994212621 podStartE2EDuration="1.994212621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:09.991905616 +0000 UTC m=+2030.424669508" watchObservedRunningTime="2026-10-30 07:31:09.994212621 +0000 UTC m=+2030.426976515"
Oct 30 07:31:10.512141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:10.512104    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:10.961593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:10.961554    6738 generic.go:356] "Generic (PLEG): container finished" podID="61b83d00-e8c7-4a35-b4f9-3c21a9c71257" containerID="8b299d060d33573221b23ad0bedc7f4b7f24c8f27746a4bbc9c861e968e591e4" exitCode=0
Oct 30 07:31:10.962155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:10.961631    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2" event={"ID":"61b83d00-e8c7-4a35-b4f9-3c21a9c71257","Type":"ContainerDied","Data":"8b299d060d33573221b23ad0bedc7f4b7f24c8f27746a4bbc9c861e968e591e4"}
Oct 30 07:31:12.137815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.137788    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:12.284809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.284711    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\") pod \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") "
Oct 30 07:31:12.284809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.284784    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\") pod \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") "
Oct 30 07:31:12.285037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.284959    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\" (UniqueName: \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\") pod \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\" (UID: \"61b83d00-e8c7-4a35-b4f9-3c21a9c71257\") "
Oct 30 07:31:12.285133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.285109    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca" pod "61b83d00-e8c7-4a35-b4f9-3c21a9c71257" (UID: "61b83d00-e8c7-4a35-b4f9-3c21a9c71257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:12.285555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.285537    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:12.287140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.287113    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca" pod "61b83d00-e8c7-4a35-b4f9-3c21a9c71257" (UID: "61b83d00-e8c7-4a35-b4f9-3c21a9c71257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:12.287140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.287129    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w" pod "61b83d00-e8c7-4a35-b4f9-3c21a9c71257" (UID: "61b83d00-e8c7-4a35-b4f9-3c21a9c71257"). InnerVolumeSpecName "kube-api-access-tqx4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:12.387127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.387085    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:12.387127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.387121    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqx4w\" (UniqueName: \"kubernetes.io/projected/61b83d00-e8c7-4a35-b4f9-3c21a9c71257-kube-api-access-tqx4w\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:12.439130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.439089    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:12.982642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.982604    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"
Oct 30 07:31:12.982847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.982639    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2" event={"ID":"61b83d00-e8c7-4a35-b4f9-3c21a9c71257","Type":"ContainerDied","Data":"4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb"}
Oct 30 07:31:12.982847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:12.982696    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4612d797cb9066d48495ef1b3b36b0cb6d9ed4aa06e232717183bb0a81d0ffcb"
Oct 30 07:31:13.109425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.109391    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:31:13.109929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.109638    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy" containerID="cri-o://46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb" gracePeriod=30
Oct 30 07:31:13.122089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.122043    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:31:13.122347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.122319    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy" containerID="cri-o://0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80" gracePeriod=30
Oct 30 07:31:13.126119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.126020    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:31:13.126755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.126560    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy" containerID="cri-o://1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7" gracePeriod=30
Oct 30 07:31:13.131177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.131144    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:31:13.131437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:13.131410    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy" containerID="cri-o://010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b" gracePeriod=30
Oct 30 07:31:14.328110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:14.328070    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:16.115011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:16.114976    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:17.779584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:17.779546    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:31:17.780069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:17.779790    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy" containerID="cri-o://29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3" gracePeriod=30
Oct 30 07:31:17.792396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:17.792305    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:31:17.792551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:17.792517    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy" containerID="cri-o://617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb" gracePeriod=30
Oct 30 07:31:18.061150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.061053    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:18.480758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.480697    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:31:18.489079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.489038    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:31:18.535630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.535605    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:31:18.537545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.537527    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:31:18.628662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628610    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.628662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628663    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628688    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628707    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628750    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628776    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628815    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628841    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628882    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628907    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628934    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.628965    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629033    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629075    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629110    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629150    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629179    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629206    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629257    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629313    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629343    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629374    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629413    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629511    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629549    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629631    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.629781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629756    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629806    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629854    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\") pod \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\" (UID: \"d7b18509-60e3-442f-b08d-d0c42cc6a6d2\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629896    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.629966    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.630012    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.630064    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.630112    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\") pod \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\" (UID: \"41f2806b-eef5-4ce8-b7ee-93c5605871b3\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.630154    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\") pod \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\" (UID: \"bc3b8adb-b136-4124-81ec-7ef4eba777a8\") "
Oct 30 07:31:18.630590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.630211    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\") pod \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\" (UID: \"65cabdbc-f6f2-4995-8a38-2501eb972c3a\") "
Oct 30 07:31:18.631701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.631669    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.631900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.631878    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.632069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.632048    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.632385    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.632798    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.633367    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.634714    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.634963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.639427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.635145    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.643770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.642659    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.643770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.642874    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.643770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.643236    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.644624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.644595    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.644921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.644896    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.646160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.645945    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.647931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.647699    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "kube-api-access-gdb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648167    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:18.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648215    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648253    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648488    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648496    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648670    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648757    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648807    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648842    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.648898    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.649230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.649127    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:18.649970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.649943    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "kube-api-access-s5wwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.650134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.650031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.650134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.650089    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.650500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.650462    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo" pod "65cabdbc-f6f2-4995-8a38-2501eb972c3a" (UID: "65cabdbc-f6f2-4995-8a38-2501eb972c3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:18.650784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.650759    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo" pod "bc3b8adb-b136-4124-81ec-7ef4eba777a8" (UID: "bc3b8adb-b136-4124-81ec-7ef4eba777a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:18.659154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.659095    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "kube-api-access-q6ftq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.660720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.660691    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo" pod "d7b18509-60e3-442f-b08d-d0c42cc6a6d2" (UID: "d7b18509-60e3-442f-b08d-d0c42cc6a6d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:18.660720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.660720    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "kube-api-access-6vkt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.662397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.662364    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo" pod "41f2806b-eef5-4ce8-b7ee-93c5605871b3" (UID: "41f2806b-eef5-4ce8-b7ee-93c5605871b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:18.736588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736550    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736586    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736599    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736611    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736625    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736635    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736646    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736657    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkt8\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-kube-api-access-6vkt8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736669    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736681    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdb98\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-kube-api-access-gdb98\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736693    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736705    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736716    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736736    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736749    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736760    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736772    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736784    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736796    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736807    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736820    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736832    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736845    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ftq\" (UniqueName: \"kubernetes.io/projected/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-kube-api-access-q6ftq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.736858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736857    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736894    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736909    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736923    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wwz\" (UniqueName: \"kubernetes.io/projected/65cabdbc-f6f2-4995-8a38-2501eb972c3a-kube-api-access-s5wwz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736936    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736950    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cabdbc-f6f2-4995-8a38-2501eb972c3a-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736963    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736977    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.736991    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3b8adb-b136-4124-81ec-7ef4eba777a8-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.737002    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cabdbc-f6f2-4995-8a38-2501eb972c3a-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.737017    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f2806b-eef5-4ce8-b7ee-93c5605871b3-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.737029    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b18509-60e3-442f-b08d-d0c42cc6a6d2-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.737667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.737043    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3b8adb-b136-4124-81ec-7ef4eba777a8-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.826619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.826582    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"]
Oct 30 07:31:18.827368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827349    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy"
Oct 30 07:31:18.827368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827365    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy"
Oct 30 07:31:18.827368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827374    6738 cpu_manager.go:414] "containerMap: removing container" podUID="61b83d00-e8c7-4a35-b4f9-3c21a9c71257" containerName="revoke-keys"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827380    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b83d00-e8c7-4a35-b4f9-3c21a9c71257" containerName="revoke-keys"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827412    6738 cpu_manager.go:414] "containerMap: removing container" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827419    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827428    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827433    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827443    6738 cpu_manager.go:414] "containerMap: removing container" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827448    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827472    6738 cpu_manager.go:414] "containerMap: removing container" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.827576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827479    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827591    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerName="istio-proxy"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827598    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b83d00-e8c7-4a35-b4f9-3c21a9c71257" containerName="revoke-keys"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827610    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerName="istio-proxy"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827618    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerName="istio-proxy"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827626    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.827863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.827634    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerName="istio-proxy"
Oct 30 07:31:18.832963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.832937    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:18.849020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.848985    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:31:18.849020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.849011    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:31:18.850334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.850303    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"]
Oct 30 07:31:18.940155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.940055    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:18.940155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.940109    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdw6\" (UniqueName: \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:18.940420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:18.940376    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.034211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.034175    6738 generic.go:356] "Generic (PLEG): container finished" podID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" containerID="46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb" exitCode=0
Oct 30 07:31:19.034424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.034246    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"
Oct 30 07:31:19.034424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.034257    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" event={"ID":"bc3b8adb-b136-4124-81ec-7ef4eba777a8","Type":"ContainerDied","Data":"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"}
Oct 30 07:31:19.034424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.034336    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f" event={"ID":"bc3b8adb-b136-4124-81ec-7ef4eba777a8","Type":"ContainerDied","Data":"631c19825cfe6d258a66ebbc7c67f27f30436db0136b4c5be6499f5d5e3f55ce"}
Oct 30 07:31:19.034424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.034365    6738 scope.go:122] "RemoveContainer" containerID="46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"
Oct 30 07:31:19.036013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.035993    6738 generic.go:356] "Generic (PLEG): container finished" podID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" containerID="0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80" exitCode=0
Oct 30 07:31:19.036093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.036028    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" event={"ID":"d7b18509-60e3-442f-b08d-d0c42cc6a6d2","Type":"ContainerDied","Data":"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"}
Oct 30 07:31:19.036093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.036071    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"
Oct 30 07:31:19.036093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.036082    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk" event={"ID":"d7b18509-60e3-442f-b08d-d0c42cc6a6d2","Type":"ContainerDied","Data":"3ff675bed729ce562852b5df29732c8c160cb3363bc27859eff673763375f660"}
Oct 30 07:31:19.038338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.038315    6738 generic.go:356] "Generic (PLEG): container finished" podID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" containerID="1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7" exitCode=0
Oct 30 07:31:19.038457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.038441    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"
Oct 30 07:31:19.038541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.038453    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerDied","Data":"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"}
Oct 30 07:31:19.038541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.038503    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz" event={"ID":"65cabdbc-f6f2-4995-8a38-2501eb972c3a","Type":"ContainerDied","Data":"6fb335c0dae4d218d491074c24151e67f72e28cfde8e29dfd81f6fe00d152abb"}
Oct 30 07:31:19.040182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.040158    6738 generic.go:356] "Generic (PLEG): container finished" podID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" containerID="010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b" exitCode=0
Oct 30 07:31:19.040342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.040210    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"
Oct 30 07:31:19.040342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.040245    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" event={"ID":"41f2806b-eef5-4ce8-b7ee-93c5605871b3","Type":"ContainerDied","Data":"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"}
Oct 30 07:31:19.040342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.040285    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf" event={"ID":"41f2806b-eef5-4ce8-b7ee-93c5605871b3","Type":"ContainerDied","Data":"dbe8f3b61ee96f38d1317f8e52bcb9f5e75b5689b8d7cfd4a80a336b82b0d21c"}
Oct 30 07:31:19.041497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.041472    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.041664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.041646    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.041739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.041686    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdw6\" (UniqueName: \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.042277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.042252    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.045006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.044982    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.049122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.049103    6738 scope.go:122] "RemoveContainer" containerID="46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"
Oct 30 07:31:19.049507 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:19.049412    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb\": container with ID starting with 46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb not found: ID does not exist" containerID="46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"
Oct 30 07:31:19.049507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.049455    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb"} err="failed to get container status \"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb\": rpc error: code = NotFound desc = could not find container \"46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb\": container with ID starting with 46eb51255c6e7e87efcb095586c882a5423dfd3c1ef0a3128a60008e21cc5cbb not found: ID does not exist"
Oct 30 07:31:19.049507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.049479    6738 scope.go:122] "RemoveContainer" containerID="0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"
Oct 30 07:31:19.060866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.060845    6738 scope.go:122] "RemoveContainer" containerID="0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"
Oct 30 07:31:19.061161 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:19.061125    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80\": container with ID starting with 0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80 not found: ID does not exist" containerID="0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"
Oct 30 07:31:19.061229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.061166    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80"} err="failed to get container status \"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80\": rpc error: code = NotFound desc = could not find container \"0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80\": container with ID starting with 0de0d5e8c377ec34224fe028d4cb631ba2699a17224ea4bce477159391fb3a80 not found: ID does not exist"
Oct 30 07:31:19.061229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.061184    6738 scope.go:122] "RemoveContainer" containerID="1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"
Oct 30 07:31:19.065195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.065168    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdw6\" (UniqueName: \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\") pod \"maas-api-revoke-sub-8e41aa52-84lgz\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.072257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.072237    6738 scope.go:122] "RemoveContainer" containerID="3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"
Oct 30 07:31:19.082466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.082448    6738 scope.go:122] "RemoveContainer" containerID="1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"
Oct 30 07:31:19.082724 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:19.082707    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7\": container with ID starting with 1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7 not found: ID does not exist" containerID="1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"
Oct 30 07:31:19.082769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.082734    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7"} err="failed to get container status \"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7\": rpc error: code = NotFound desc = could not find container \"1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7\": container with ID starting with 1851612d26221f561d6b735359752c395d5ea6f47383c6cf78326643053043f7 not found: ID does not exist"
Oct 30 07:31:19.082769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.082751    6738 scope.go:122] "RemoveContainer" containerID="3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"
Oct 30 07:31:19.082946 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:19.082928    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f\": container with ID starting with 3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f not found: ID does not exist" containerID="3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"
Oct 30 07:31:19.082992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.082953    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f"} err="failed to get container status \"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f\": rpc error: code = NotFound desc = could not find container \"3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f\": container with ID starting with 3258f3381c16a665aa1f1d6bfacb4c0eacc5fa5c31d554947289c176142e9c1f not found: ID does not exist"
Oct 30 07:31:19.082992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.082966    6738 scope.go:122] "RemoveContainer" containerID="010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"
Oct 30 07:31:19.092834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.092811    6738 scope.go:122] "RemoveContainer" containerID="010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"
Oct 30 07:31:19.093169 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:19.093147    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b\": container with ID starting with 010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b not found: ID does not exist" containerID="010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"
Oct 30 07:31:19.093241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.093174    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b"} err="failed to get container status \"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b\": rpc error: code = NotFound desc = could not find container \"010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b\": container with ID starting with 010c6c6d50e87b9f0f40127401b7d631405243e57c20c3ee3ef38582cb0e142b not found: ID does not exist"
Oct 30 07:31:19.135774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.135736    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:31:19.140076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.140047    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4kxtf"]
Oct 30 07:31:19.150919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.150893    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:19.181002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.180965    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:31:19.184937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.184905    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-w84wk"]
Oct 30 07:31:19.225424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.223719    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:31:19.236987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.236956    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-lbqhz"]
Oct 30 07:31:19.267388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.267351    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:31:19.275796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.275759    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-f7p4f"]
Oct 30 07:31:19.474118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.474050    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"]
Oct 30 07:31:19.474638 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:19.474608    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10bcbeb_3b94_41b7_a28f_ccb8cc4de118.slice/crio-1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac WatchSource:0}: Error finding container 1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac: Status 404 returned error can't find the container with id 1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac
Oct 30 07:31:19.694095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.694060    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f2806b-eef5-4ce8-b7ee-93c5605871b3" path="/var/lib/kubelet/pods/41f2806b-eef5-4ce8-b7ee-93c5605871b3/volumes"
Oct 30 07:31:19.694753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.694734    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cabdbc-f6f2-4995-8a38-2501eb972c3a" path="/var/lib/kubelet/pods/65cabdbc-f6f2-4995-8a38-2501eb972c3a/volumes"
Oct 30 07:31:19.695356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.695338    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3b8adb-b136-4124-81ec-7ef4eba777a8" path="/var/lib/kubelet/pods/bc3b8adb-b136-4124-81ec-7ef4eba777a8/volumes"
Oct 30 07:31:19.696150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.696127    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b18509-60e3-442f-b08d-d0c42cc6a6d2" path="/var/lib/kubelet/pods/d7b18509-60e3-442f-b08d-d0c42cc6a6d2/volumes"
Oct 30 07:31:19.881207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.881181    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 07:31:19.886616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.886594    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:19.903941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.903915    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d09eb6-serving-cert\""
Oct 30 07:31:19.905963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.905933    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 07:31:19.953930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.953898    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6k2\" (UniqueName: \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:19.954088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.954035    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:19.954133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.954117    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:19.954232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.954215    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:19.988033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:19.988002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:20.057030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.056988    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6k2\" (UniqueName: \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.057205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.057117    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.057284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.057214    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.057555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.057279    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.058531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.058482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.060351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.060285    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.060629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.060603    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.064609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.064578    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz" event={"ID":"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118","Type":"ContainerStarted","Data":"1c99d01c222bf65fff58890edc33817058b12e89f26f626281670ecf8eda18e1"}
Oct 30 07:31:20.064716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.064668    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz" event={"ID":"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118","Type":"ContainerStarted","Data":"1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac"}
Oct 30 07:31:20.081158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.081128    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6k2\" (UniqueName: \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\") pod \"maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.105818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.105761    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz" podStartSLOduration=2.105748935 podStartE2EDuration="2.105748935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:20.105230527 +0000 UTC m=+2040.537994422" watchObservedRunningTime="2026-10-30 07:31:20.105748935 +0000 UTC m=+2040.538512826"
Oct 30 07:31:20.114537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.114504    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:20.208370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.205418    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:20.232347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.232271    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:20.546402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.546373    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 07:31:20.583556 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:20.583522    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f92c2c_e3f2_4cd4_a4cf_d08cfeb6ae11.slice/crio-fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0 WatchSource:0}: Error finding container fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0: Status 404 returned error can't find the container with id fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0
Oct 30 07:31:20.585102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:20.585081    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:31:21.108320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.100287    6738 generic.go:356] "Generic (PLEG): container finished" podID="a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" containerID="1c99d01c222bf65fff58890edc33817058b12e89f26f626281670ecf8eda18e1" exitCode=0
Oct 30 07:31:21.108320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.100596    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz" event={"ID":"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118","Type":"ContainerDied","Data":"1c99d01c222bf65fff58890edc33817058b12e89f26f626281670ecf8eda18e1"}
Oct 30 07:31:21.127319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.125961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" event={"ID":"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11","Type":"ContainerStarted","Data":"17035223141b959069c25c4c0094e1fdc54f83dd1006e2a20ea36a89f1c2f9d5"}
Oct 30 07:31:21.127319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.126036    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" event={"ID":"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11","Type":"ContainerStarted","Data":"fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0"}
Oct 30 07:31:21.127319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.127005    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:21.204506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:21.204382    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" podStartSLOduration=1.880024666 podStartE2EDuration="2.204361539s" totalImagesPullingTime="324.336873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:21.199759909 +0000 UTC m=+2041.632523814" watchObservedRunningTime="2026-10-30 07:31:21.204361539 +0000 UTC m=+2041.637125436"
Oct 30 07:31:22.055330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.055259    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:22.313140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.313078    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:22.432925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.432889    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\") pod \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") "
Oct 30 07:31:22.433134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.433115    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\" (UniqueName: \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\") pod \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") "
Oct 30 07:31:22.433210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.433195    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\") pod \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\" (UID: \"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118\") "
Oct 30 07:31:22.433384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.433360    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca" pod "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" (UID: "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:22.433748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.433727    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:22.435268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.435240    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca" pod "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" (UID: "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:22.435401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.435318    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6" pod "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" (UID: "a10bcbeb-3b94-41b7-a28f-ccb8cc4de118"). InnerVolumeSpecName "kube-api-access-tqdw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:22.535327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.535264    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqdw6\" (UniqueName: \"kubernetes.io/projected/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-kube-api-access-tqdw6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:22.535327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:22.535325    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.053361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.053234    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:31:23.110998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.110970    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:31:23.145429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145387    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145514    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145602    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145667    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145806    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145844    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145904    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.145971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145941    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.146144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.145972    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\") pod \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\" (UID: \"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f\") "
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.146682    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.147011    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.147042    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.147166    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.147212    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.147307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.147275    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.149091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.149061    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.149989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.149959    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "kube-api-access-qwgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.152121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.152092    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.152355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.152286    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo" pod "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" (UID: "a0b5ac4e-fa12-4488-afc3-f3a92b76f47f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:23.179156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.175252    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"
Oct 30 07:31:23.179156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.175530    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" event={"ID":"bb7def7b-3beb-47b2-9d5c-8aa52035eea8","Type":"ContainerDied","Data":"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"}
Oct 30 07:31:23.179156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.175598    6738 scope.go:122] "RemoveContainer" containerID="617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"
Oct 30 07:31:23.179156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.175158    6738 generic.go:356] "Generic (PLEG): container finished" podID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerID="617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb" exitCode=0
Oct 30 07:31:23.179156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.175894    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2" event={"ID":"bb7def7b-3beb-47b2-9d5c-8aa52035eea8","Type":"ContainerDied","Data":"0dce5e60132f89bd2a1cc86f72d54275e68389e608d5b31ea116d58eec084591"}
Oct 30 07:31:23.180246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.180089    6738 generic.go:356] "Generic (PLEG): container finished" podID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerID="29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3" exitCode=0
Oct 30 07:31:23.180952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.180534    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" event={"ID":"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f","Type":"ContainerDied","Data":"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"}
Oct 30 07:31:23.180952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.180586    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl" event={"ID":"a0b5ac4e-fa12-4488-afc3-f3a92b76f47f","Type":"ContainerDied","Data":"924480d8faa759c01870b9d181231535a757f782677e32dffdc3c093e137e0c9"}
Oct 30 07:31:23.180952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.180664    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"
Oct 30 07:31:23.183914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.183883    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"
Oct 30 07:31:23.185116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.185062    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz" event={"ID":"a10bcbeb-3b94-41b7-a28f-ccb8cc4de118","Type":"ContainerDied","Data":"1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac"}
Oct 30 07:31:23.185116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.185109    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d16b3739f6e5c0a49d70ba43959989dc2bdaa4e92acbca709ee2229648bd9ac"
Oct 30 07:31:23.207930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.207900    6738 scope.go:122] "RemoveContainer" containerID="617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"
Oct 30 07:31:23.208331 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:23.208280    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb\": container with ID starting with 617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb not found: ID does not exist" containerID="617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"
Oct 30 07:31:23.208432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.208346    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb"} err="failed to get container status \"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb\": rpc error: code = NotFound desc = could not find container \"617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb\": container with ID starting with 617c786a5681f67b361c825e7502a9e82a06954156e3e48daf0ef19168caeceb not found: ID does not exist"
Oct 30 07:31:23.208432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.208370    6738 scope.go:122] "RemoveContainer" containerID="29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"
Oct 30 07:31:23.224656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.221181    6738 scope.go:122] "RemoveContainer" containerID="29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"
Oct 30 07:31:23.224656 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:23.221909    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3\": container with ID starting with 29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3 not found: ID does not exist" containerID="29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"
Oct 30 07:31:23.224656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.221951    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3"} err="failed to get container status \"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3\": rpc error: code = NotFound desc = could not find container \"29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3\": container with ID starting with 29356acd915e3b18c4d3b03980e114e7343dd7b2bc20f9027dc8772c6df24ea3 not found: ID does not exist"
Oct 30 07:31:23.247762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.247726    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.247762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.247767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.247916    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.247954    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248062    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.248151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248099    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248161    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248203    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248309    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\") pod \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\" (UID: \"bb7def7b-3beb-47b2-9d5c-8aa52035eea8\") "
Oct 30 07:31:23.248352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248273    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.248561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248440    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.248561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248474    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:23.249005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.248977    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249004    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249017    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249029    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249040    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249052    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgk5\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-kube-api-access-qwgk5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249065    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249078    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249090    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249101    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249107    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249113    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.249166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.249158    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.250809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.250788    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.250920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.250848    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "kube-api-access-dggwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.250920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.250885    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.251143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.251126    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo" pod "bb7def7b-3beb-47b2-9d5c-8aa52035eea8" (UID: "bb7def7b-3beb-47b2-9d5c-8aa52035eea8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:23.266661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.266627    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:31:23.269935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.269905    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-vrlpl"]
Oct 30 07:31:23.350256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.350162    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.350256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.350192    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.350256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.350205    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dggwl\" (UniqueName: \"kubernetes.io/projected/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-kube-api-access-dggwl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.350256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.350218    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.350256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.350248    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7def7b-3beb-47b2-9d5c-8aa52035eea8-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.535416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.535372    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:31:23.537637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.537603    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-q2gn2"]
Oct 30 07:31:23.691070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.691033    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" path="/var/lib/kubelet/pods/a0b5ac4e-fa12-4488-afc3-f3a92b76f47f/volumes"
Oct 30 07:31:23.691573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.691559    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" path="/var/lib/kubelet/pods/bb7def7b-3beb-47b2-9d5c-8aa52035eea8/volumes"
Oct 30 07:31:23.936377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:23.930031    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:25.802355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:25.802320    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:27.558358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:27.558315    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:28.189233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.189206    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 07:31:28.597619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.597513    6738 scope.go:122] "RemoveContainer" containerID="68db72e00ae6b8215b3294e8db9eaff675addcd62d1c4f881d9c21c6f0a68ae7"
Oct 30 07:31:28.617018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.616878    6738 scope.go:122] "RemoveContainer" containerID="8ce921f165e52b3b3b6f6e58ed3f5e1606816cb1d2fe10114ca5d69d5856e903"
Oct 30 07:31:28.627531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.627500    6738 scope.go:122] "RemoveContainer" containerID="36c17d1dc8b58ba97f1309f7fa5a96e32f3828d03a74dfb24d729dea62e7ab66"
Oct 30 07:31:28.637662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.637643    6738 scope.go:122] "RemoveContainer" containerID="d5cf672c4841f2514bc7b90ab68e69f54c65bda62e2ad036440a19c0f6f9646c"
Oct 30 07:31:28.648590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.648567    6738 scope.go:122] "RemoveContainer" containerID="eb8215e441667769622c792bb8098521f7b1dba45467c8f0471376b9aa143e2e"
Oct 30 07:31:28.658145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.658120    6738 scope.go:122] "RemoveContainer" containerID="b4d6a32d8f51ab327c1902cd9cd5551a2c370f52a2206fc76f89f40debf740f1"
Oct 30 07:31:28.667126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.667102    6738 scope.go:122] "RemoveContainer" containerID="2db7890e0e597871838a48946ea6c36334ac0cbcd2131ff87eb9ca92b46ac50e"
Oct 30 07:31:28.676622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.676600    6738 scope.go:122] "RemoveContainer" containerID="18e33d7c4e923038a92678beb779931642b74d54a3e3fdd3698f586b615bc837"
Oct 30 07:31:28.686136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.686117    6738 scope.go:122] "RemoveContainer" containerID="4b03c8d9ccadff45fdecd74924c6a555ab4e55f22f20d14aa1bb83a6d7d6c3f9"
Oct 30 07:31:28.695702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.695680    6738 scope.go:122] "RemoveContainer" containerID="4b8df58444ec265a268975198300a2b39ba990eb5f30b722012048c2c953628c"
Oct 30 07:31:28.705413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.705390    6738 scope.go:122] "RemoveContainer" containerID="fd2005044992417b74d5e5ef6b530bebb6f5bb3bde8b1c2deb6d4a02cc0b467d"
Oct 30 07:31:28.716952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:28.716930    6738 scope.go:122] "RemoveContainer" containerID="42acfe19597c9ee782db08a39cd771415e9b931c9eefa2087eb6f0414296b1db"
Oct 30 07:31:29.352041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:29.352003    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:31.144274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:31.144240    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:31.261698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:31.261444    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:31.713190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:31.713152    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:33.088871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:33.088828    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:35.041316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:35.041268    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:36.813654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:36.813620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:37.528510 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:37.528467    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:31:38.603834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:38.603792    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:40.162565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.162507    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:31:40.210436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.210402    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:40.453427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.447216    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-5491f3"]
Oct 30 07:31:40.457227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.454664    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy"
Oct 30 07:31:40.457494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.457467    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy"
Oct 30 07:31:40.457604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.457593    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy"
Oct 30 07:31:40.457713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.457701    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy"
Oct 30 07:31:40.457847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.457838    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" containerName="revoke-keys"
Oct 30 07:31:40.457917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.457908    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" containerName="revoke-keys"
Oct 30 07:31:40.458248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.458230    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" containerName="revoke-keys"
Oct 30 07:31:40.458422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.458408    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b5ac4e-fa12-4488-afc3-f3a92b76f47f" containerName="istio-proxy"
Oct 30 07:31:40.458516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.458505    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7def7b-3beb-47b2-9d5c-8aa52035eea8" containerName="istio-proxy"
Oct 30 07:31:40.470032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.469002    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:40.485080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.485046    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-5491f3"]
Oct 30 07:31:40.486153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.485584    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5q8j4\""
Oct 30 07:31:40.584741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.584683    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnwh\" (UniqueName: \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\") pod \"test-curl-7436-5491f3\" (UID: \"923a672e-146d-4ebd-acd0-9813549437ad\") " pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:40.685379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.685343    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:40.685828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.685804    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnwh\" (UniqueName: \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\") pod \"test-curl-7436-5491f3\" (UID: \"923a672e-146d-4ebd-acd0-9813549437ad\") " pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:40.709177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.709107    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnwh\" (UniqueName: \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\") pod \"test-curl-7436-5491f3\" (UID: \"923a672e-146d-4ebd-acd0-9813549437ad\") " pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:40.808532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:40.808500    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:41.124080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:41.124053    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-5491f3"]
Oct 30 07:31:41.124887 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:41.124856    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod923a672e_146d_4ebd_acd0_9813549437ad.slice/crio-ba0ca8d9c7471444c28a59d6899f3901837d3c2a33a9a9c88ba7480f2e2b953c WatchSource:0}: Error finding container ba0ca8d9c7471444c28a59d6899f3901837d3c2a33a9a9c88ba7480f2e2b953c: Status 404 returned error can't find the container with id ba0ca8d9c7471444c28a59d6899f3901837d3c2a33a9a9c88ba7480f2e2b953c
Oct 30 07:31:41.366510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:41.366474    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-5491f3" event={"ID":"923a672e-146d-4ebd-acd0-9813549437ad","Type":"ContainerStarted","Data":"f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3"}
Oct 30 07:31:41.366962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:41.366517    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-5491f3" event={"ID":"923a672e-146d-4ebd-acd0-9813549437ad","Type":"ContainerStarted","Data":"ba0ca8d9c7471444c28a59d6899f3901837d3c2a33a9a9c88ba7480f2e2b953c"}
Oct 30 07:31:41.407758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:41.407699    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7436-5491f3" podStartSLOduration=1.407684245 podStartE2EDuration="1.407684245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:41.406544665 +0000 UTC m=+2061.839308557" watchObservedRunningTime="2026-10-30 07:31:41.407684245 +0000 UTC m=+2061.840448159"
Oct 30 07:31:42.393176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.393078    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-5491f3"]
Oct 30 07:31:42.398411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.398381    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-5491f3"]
Oct 30 07:31:42.555250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.555215    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-70a5f7"]
Oct 30 07:31:42.555936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.555921    6738 cpu_manager.go:414] "containerMap: removing container" podUID="923a672e-146d-4ebd-acd0-9813549437ad" containerName="test-curl-7436-5491f3"
Oct 30 07:31:42.555993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.555938    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923a672e-146d-4ebd-acd0-9813549437ad" containerName="test-curl-7436-5491f3"
Oct 30 07:31:42.556097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.556087    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="923a672e-146d-4ebd-acd0-9813549437ad" containerName="test-curl-7436-5491f3"
Oct 30 07:31:42.559713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.559693    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:42.573455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.573422    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-70a5f7"]
Oct 30 07:31:42.616375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.616336    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2w9l\" (UniqueName: \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\") pod \"test-curl-7436-70a5f7\" (UID: \"00209875-65ae-4e52-8cf8-b8a93873986a\") " pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:42.655721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.655612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:42.728330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.725166    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2w9l\" (UniqueName: \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\") pod \"test-curl-7436-70a5f7\" (UID: \"00209875-65ae-4e52-8cf8-b8a93873986a\") " pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:42.759543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.759513    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2w9l\" (UniqueName: \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\") pod \"test-curl-7436-70a5f7\" (UID: \"00209875-65ae-4e52-8cf8-b8a93873986a\") " pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:42.788489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.788457    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7433-652852"]
Oct 30 07:31:42.793191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.793164    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:42.808578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.808545    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7433-652852"]
Oct 30 07:31:42.875848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.875815    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:42.932640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:42.932607    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgfd\" (UniqueName: \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\") pod \"test-curl-7433-652852\" (UID: \"042f074d-5258-48d8-ba14-984d23ad5965\") " pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:43.033730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.033693    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgfd\" (UniqueName: \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\") pod \"test-curl-7433-652852\" (UID: \"042f074d-5258-48d8-ba14-984d23ad5965\") " pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:43.057184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.057154    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgfd\" (UniqueName: \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\") pod \"test-curl-7433-652852\" (UID: \"042f074d-5258-48d8-ba14-984d23ad5965\") " pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:43.107737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.107700    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:43.182359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.182277    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-70a5f7"]
Oct 30 07:31:43.184890 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:43.183414    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00209875_65ae_4e52_8cf8_b8a93873986a.slice/crio-1f4215154a534618684aedf6a3062adef1399a301d34708b9b66bbfc1958939e WatchSource:0}: Error finding container 1f4215154a534618684aedf6a3062adef1399a301d34708b9b66bbfc1958939e: Status 404 returned error can't find the container with id 1f4215154a534618684aedf6a3062adef1399a301d34708b9b66bbfc1958939e
Oct 30 07:31:43.399172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.399129    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-70a5f7" event={"ID":"00209875-65ae-4e52-8cf8-b8a93873986a","Type":"ContainerStarted","Data":"11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599"}
Oct 30 07:31:43.399628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.399186    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-70a5f7" event={"ID":"00209875-65ae-4e52-8cf8-b8a93873986a","Type":"ContainerStarted","Data":"1f4215154a534618684aedf6a3062adef1399a301d34708b9b66bbfc1958939e"}
Oct 30 07:31:43.399628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.399422    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7436-5491f3" podUID="923a672e-146d-4ebd-acd0-9813549437ad" containerName="test-curl-7436-5491f3" containerID="cri-o://f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3" gracePeriod=2
Oct 30 07:31:43.413518 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:43.413486    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042f074d_5258_48d8_ba14_984d23ad5965.slice/crio-240b27e6a995394586f4b65d6f08a37589d8b5d0d890b742081f60f00c91613f WatchSource:0}: Error finding container 240b27e6a995394586f4b65d6f08a37589d8b5d0d890b742081f60f00c91613f: Status 404 returned error can't find the container with id 240b27e6a995394586f4b65d6f08a37589d8b5d0d890b742081f60f00c91613f
Oct 30 07:31:43.413941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.413914    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7433-652852"]
Oct 30 07:31:43.440146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.440058    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7436-70a5f7" podStartSLOduration=1.4400437529999999 podStartE2EDuration="1.440043753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:43.43737357 +0000 UTC m=+2063.870137465" watchObservedRunningTime="2026-10-30 07:31:43.440043753 +0000 UTC m=+2063.872807646"
Oct 30 07:31:43.451154 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:43.451117    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-5491f3\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="923a672e-146d-4ebd-acd0-9813549437ad" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:43.515839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.515807    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"]
Oct 30 07:31:43.520735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.520702    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.534761 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:43.534722    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-5491f3\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="923a672e-146d-4ebd-acd0-9813549437ad" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:43.537914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.537883    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:31:43.538147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.538116    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"]
Oct 30 07:31:43.538288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.538163    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:31:43.643037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.643001    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.643230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.643070    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2rq\" (UniqueName: \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.643230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.643179    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.745069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.744974    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.745069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.745053    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2rq\" (UniqueName: \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.745344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.745098    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.745893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.745869    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.747546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.747524    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.769551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.769515    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2rq\" (UniqueName: \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\") pod \"maas-api-revoke-sub-a019a3e9-5mqls\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:43.836033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:43.836002    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:44.143390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.143361    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"]
Oct 30 07:31:44.144180 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:44.144158    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1d4e4d_79ef_42b0_a3d0_a7d8af86899d.slice/crio-223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b WatchSource:0}: Error finding container 223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b: Status 404 returned error can't find the container with id 223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b
Oct 30 07:31:44.407883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.407783    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls" event={"ID":"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d","Type":"ContainerStarted","Data":"223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b"}
Oct 30 07:31:44.409353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.409326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7433-652852" event={"ID":"042f074d-5258-48d8-ba14-984d23ad5965","Type":"ContainerStarted","Data":"2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7"}
Oct 30 07:31:44.409562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.409360    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7433-652852" event={"ID":"042f074d-5258-48d8-ba14-984d23ad5965","Type":"ContainerStarted","Data":"240b27e6a995394586f4b65d6f08a37589d8b5d0d890b742081f60f00c91613f"}
Oct 30 07:31:44.421339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.421277    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-70a5f7"]
Oct 30 07:31:44.425131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.425102    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-70a5f7"]
Oct 30 07:31:44.451467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.451403    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7433-652852" podStartSLOduration=2.451382708 podStartE2EDuration="2.451382708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:44.447814534 +0000 UTC m=+2064.880578429" watchObservedRunningTime="2026-10-30 07:31:44.451382708 +0000 UTC m=+2064.884146603"
Oct 30 07:31:44.545608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.545573    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:44.630487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.628151    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-826d85"]
Oct 30 07:31:44.630487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.629255    6738 cpu_manager.go:414] "containerMap: removing container" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" containerName="test-curl-7436-70a5f7"
Oct 30 07:31:44.630487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.629288    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" containerName="test-curl-7436-70a5f7"
Oct 30 07:31:44.632144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.630907    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" containerName="test-curl-7436-70a5f7"
Oct 30 07:31:44.636353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.636331    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:44.658673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.658600    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-826d85"]
Oct 30 07:31:44.758481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.758445    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwlf\" (UniqueName: \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\") pod \"test-curl-7436-826d85\" (UID: \"bab8cff0-6349-46bd-b975-1ccad6447144\") " pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:44.860395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.860354    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwlf\" (UniqueName: \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\") pod \"test-curl-7436-826d85\" (UID: \"bab8cff0-6349-46bd-b975-1ccad6447144\") " pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:44.884279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.884248    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwlf\" (UniqueName: \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\") pod \"test-curl-7436-826d85\" (UID: \"bab8cff0-6349-46bd-b975-1ccad6447144\") " pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:44.951089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:44.951006    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:45.257321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.257281    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-826d85"]
Oct 30 07:31:45.257646 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:45.257620    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab8cff0_6349_46bd_b975_1ccad6447144.slice/crio-64ed8c344841456d7d9825b115f3236fc83f6e8dd7b9c957d1b2900e595ed633 WatchSource:0}: Error finding container 64ed8c344841456d7d9825b115f3236fc83f6e8dd7b9c957d1b2900e595ed633: Status 404 returned error can't find the container with id 64ed8c344841456d7d9825b115f3236fc83f6e8dd7b9c957d1b2900e595ed633
Oct 30 07:31:45.418127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.418092    6738 generic.go:356] "Generic (PLEG): container finished" podID="8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" containerID="bb06ff45edbfec357de935168ba6f1e3fabaa42ab490e751c34c7628339fee76" exitCode=0
Oct 30 07:31:45.418611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.418163    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls" event={"ID":"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d","Type":"ContainerDied","Data":"bb06ff45edbfec357de935168ba6f1e3fabaa42ab490e751c34c7628339fee76"}
Oct 30 07:31:45.420039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.420016    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-826d85" event={"ID":"bab8cff0-6349-46bd-b975-1ccad6447144","Type":"ContainerStarted","Data":"7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d"}
Oct 30 07:31:45.420170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.420045    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-826d85" event={"ID":"bab8cff0-6349-46bd-b975-1ccad6447144","Type":"ContainerStarted","Data":"64ed8c344841456d7d9825b115f3236fc83f6e8dd7b9c957d1b2900e595ed633"}
Oct 30 07:31:45.420264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.420217    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7436-70a5f7" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" containerName="test-curl-7436-70a5f7" containerID="cri-o://11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599" gracePeriod=2
Oct 30 07:31:45.461385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.461356    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7433-652852"]
Oct 30 07:31:45.474286 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:45.472481    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-70a5f7\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:45.474286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.472516    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7433-652852"]
Oct 30 07:31:45.514529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.514400    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7436-826d85" podStartSLOduration=1.514380975 podStartE2EDuration="1.514380975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:45.51080767 +0000 UTC m=+2065.943571563" watchObservedRunningTime="2026-10-30 07:31:45.514380975 +0000 UTC m=+2065.947144868"
Oct 30 07:31:45.627395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.627367    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:45.641273 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:45.641236    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-70a5f7\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:45.672270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.672238    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\" (UniqueName: \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\") pod \"923a672e-146d-4ebd-acd0-9813549437ad\" (UID: \"923a672e-146d-4ebd-acd0-9813549437ad\") "
Oct 30 07:31:45.674409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.674379    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh" pod "923a672e-146d-4ebd-acd0-9813549437ad" (UID: "923a672e-146d-4ebd-acd0-9813549437ad"). InnerVolumeSpecName "kube-api-access-gtnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:45.694808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.694771    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923a672e-146d-4ebd-acd0-9813549437ad" path="/var/lib/kubelet/pods/923a672e-146d-4ebd-acd0-9813549437ad/volumes"
Oct 30 07:31:45.711816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.711785    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"]
Oct 30 07:31:45.712820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.712792    6738 cpu_manager.go:414] "containerMap: removing container" podUID="042f074d-5258-48d8-ba14-984d23ad5965" containerName="test-curl-7433-652852"
Oct 30 07:31:45.712820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.712817    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042f074d-5258-48d8-ba14-984d23ad5965" containerName="test-curl-7433-652852"
Oct 30 07:31:45.713016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.713001    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="042f074d-5258-48d8-ba14-984d23ad5965" containerName="test-curl-7433-652852"
Oct 30 07:31:45.717262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.717235    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.731919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.731890    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"]
Oct 30 07:31:45.780528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.780488    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.780786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.780759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txm54\" (UniqueName: \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.781460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.780801    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.781460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.780874    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnwh\" (UniqueName: \"kubernetes.io/projected/923a672e-146d-4ebd-acd0-9813549437ad-kube-api-access-gtnwh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:45.881774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.881740    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txm54\" (UniqueName: \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.881774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.881778    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.882004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.881912    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.882577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.882552    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.884147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.884127    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:45.905562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:45.905532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txm54\" (UniqueName: \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\") pod \"maas-api-revoke-sub-29ffa48c-ptpb6\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:46.032654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.032571    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:46.345772 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:46.345739    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e34c02b_b2bf_41ef_b34b_5f24194a9ab1.slice/crio-be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773 WatchSource:0}: Error finding container be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773: Status 404 returned error can't find the container with id be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773
Oct 30 07:31:46.346088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.346064    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"]
Oct 30 07:31:46.428539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.428506    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6" event={"ID":"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1","Type":"ContainerStarted","Data":"be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773"}
Oct 30 07:31:46.429950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.429921    6738 generic.go:356] "Generic (PLEG): container finished" podID="923a672e-146d-4ebd-acd0-9813549437ad" containerID="f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3" exitCode=137
Oct 30 07:31:46.430097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.429970    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-5491f3"
Oct 30 07:31:46.430097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.430022    6738 scope.go:122] "RemoveContainer" containerID="f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3"
Oct 30 07:31:46.430204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.430180    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7433-652852" podUID="042f074d-5258-48d8-ba14-984d23ad5965" containerName="test-curl-7433-652852" containerID="cri-o://2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7" gracePeriod=2
Oct 30 07:31:46.440619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.440592    6738 scope.go:122] "RemoveContainer" containerID="f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3"
Oct 30 07:31:46.440944 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:46.440925    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3\": container with ID starting with f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3 not found: ID does not exist" containerID="f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3"
Oct 30 07:31:46.441010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.440963    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3"} err="failed to get container status \"f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3\": rpc error: code = NotFound desc = could not find container \"f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3\": container with ID starting with f2a3742b555b3df1d16da94da2779e7caadc4373d28c60c5c0aaf54a19e965f3 not found: ID does not exist"
Oct 30 07:31:46.453763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.453729    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-826d85"]
Oct 30 07:31:46.457202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.457167    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-826d85"]
Oct 30 07:31:46.471345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.470563    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:46.632957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.632921    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7436-36bdb4"]
Oct 30 07:31:46.633626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.633606    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" containerName="test-curl-7436-826d85"
Oct 30 07:31:46.633626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.633624    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" containerName="test-curl-7436-826d85"
Oct 30 07:31:46.633813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.633752    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" containerName="test-curl-7436-826d85"
Oct 30 07:31:46.637281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.637255    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:46.654078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.654016    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-36bdb4"]
Oct 30 07:31:46.687610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.687579    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:46.698128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.698094    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkns\" (UniqueName: \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\") pod \"test-curl-7436-36bdb4\" (UID: \"b488a363-5579-46c0-af4d-e35e0fa91710\") " pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:46.798823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.798792    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\" (UniqueName: \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\") pod \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") "
Oct 30 07:31:46.799012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.798837    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\") pod \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") "
Oct 30 07:31:46.799012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.798983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\") pod \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\" (UID: \"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d\") "
Oct 30 07:31:46.799392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.799366    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca" pod "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" (UID: "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:46.799546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.799408    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkns\" (UniqueName: \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\") pod \"test-curl-7436-36bdb4\" (UID: \"b488a363-5579-46c0-af4d-e35e0fa91710\") " pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:46.800062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.799919    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.801093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.801069    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq" pod "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" (UID: "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d"). InnerVolumeSpecName "kube-api-access-kz2rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:46.801185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.801075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca" pod "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" (UID: "8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:46.823713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.823683    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkns\" (UniqueName: \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\") pod \"test-curl-7436-36bdb4\" (UID: \"b488a363-5579-46c0-af4d-e35e0fa91710\") " pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:46.901216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.901178    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2rq\" (UniqueName: \"kubernetes.io/projected/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-kube-api-access-kz2rq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.901216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.901211    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.954742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:46.954708    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:47.259484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.259452    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7436-36bdb4"]
Oct 30 07:31:47.260511 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:47.260482    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb488a363_5579_46c0_af4d_e35e0fa91710.slice/crio-219ea69949938277efbcf59197517135b15647245518cf2d7b75169c1566bbb3 WatchSource:0}: Error finding container 219ea69949938277efbcf59197517135b15647245518cf2d7b75169c1566bbb3: Status 404 returned error can't find the container with id 219ea69949938277efbcf59197517135b15647245518cf2d7b75169c1566bbb3
Oct 30 07:31:47.441779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.441746    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-36bdb4" event={"ID":"b488a363-5579-46c0-af4d-e35e0fa91710","Type":"ContainerStarted","Data":"522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468"}
Oct 30 07:31:47.442263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.441789    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7436-36bdb4" event={"ID":"b488a363-5579-46c0-af4d-e35e0fa91710","Type":"ContainerStarted","Data":"219ea69949938277efbcf59197517135b15647245518cf2d7b75169c1566bbb3"}
Oct 30 07:31:47.443673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.443646    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"
Oct 30 07:31:47.443794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.443683    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls" event={"ID":"8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d","Type":"ContainerDied","Data":"223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b"}
Oct 30 07:31:47.443794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.443723    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="223079d9a59bc2162033d596daa0bd39d7216e8393293b8d0d34268a99c0091b"
Oct 30 07:31:47.445620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.445591    6738 generic.go:356] "Generic (PLEG): container finished" podID="2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" containerID="7f7e142dcde093b57bc265e3dca13e251fffa61c0347ae56537378e85b559c92" exitCode=0
Oct 30 07:31:47.445707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.445665    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6" event={"ID":"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1","Type":"ContainerDied","Data":"7f7e142dcde093b57bc265e3dca13e251fffa61c0347ae56537378e85b559c92"}
Oct 30 07:31:47.445820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.445797    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7436-826d85" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" containerName="test-curl-7436-826d85" containerID="cri-o://7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d" gracePeriod=2
Oct 30 07:31:47.510078 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:47.509996    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-826d85\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:47.565375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.560418    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7436-36bdb4" podStartSLOduration=1.560395206 podStartE2EDuration="1.560395206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:47.480922387 +0000 UTC m=+2067.913686280" watchObservedRunningTime="2026-10-30 07:31:47.560395206 +0000 UTC m=+2067.993159100"
Oct 30 07:31:47.620664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.620637    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:47.634875 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:47.634833    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-826d85\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:47.711488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.711450    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\" (UniqueName: \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\") pod \"00209875-65ae-4e52-8cf8-b8a93873986a\" (UID: \"00209875-65ae-4e52-8cf8-b8a93873986a\") "
Oct 30 07:31:47.713617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.713587    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l" pod "00209875-65ae-4e52-8cf8-b8a93873986a" (UID: "00209875-65ae-4e52-8cf8-b8a93873986a"). InnerVolumeSpecName "kube-api-access-t2w9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:47.813341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:47.813305    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2w9l\" (UniqueName: \"kubernetes.io/projected/00209875-65ae-4e52-8cf8-b8a93873986a-kube-api-access-t2w9l\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:48.365075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.365045    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:48.438322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.424367    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7436-36bdb4"]
Oct 30 07:31:48.452592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.452549    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7436-36bdb4"]
Oct 30 07:31:48.472112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.472068    6738 generic.go:356] "Generic (PLEG): container finished" podID="00209875-65ae-4e52-8cf8-b8a93873986a" containerID="11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599" exitCode=137
Oct 30 07:31:48.473253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.473092    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-70a5f7"
Oct 30 07:31:48.473680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.473660    6738 scope.go:122] "RemoveContainer" containerID="11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599"
Oct 30 07:31:48.518145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.517809    6738 scope.go:122] "RemoveContainer" containerID="11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599"
Oct 30 07:31:48.518942 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:48.518834    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599\": container with ID starting with 11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599 not found: ID does not exist" containerID="11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599"
Oct 30 07:31:48.518942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.518876    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599"} err="failed to get container status \"11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599\": rpc error: code = NotFound desc = could not find container \"11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599\": container with ID starting with 11ea333300fbe139a56d423a9a95a0df7783487a5a462c64845ae48ff5858599 not found: ID does not exist"
Oct 30 07:31:48.685971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.685941    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:48.688767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.688745    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:48.831745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.831707    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\" (UniqueName: \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\") pod \"042f074d-5258-48d8-ba14-984d23ad5965\" (UID: \"042f074d-5258-48d8-ba14-984d23ad5965\") "
Oct 30 07:31:48.831927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.831797    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\" (UniqueName: \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\") pod \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") "
Oct 30 07:31:48.832041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.832024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\") pod \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") "
Oct 30 07:31:48.832135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.832120    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\") pod \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\" (UID: \"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1\") "
Oct 30 07:31:48.832525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.832484    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca" pod "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" (UID: "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:48.833018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.832993    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:48.834233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.834203    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54" pod "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" (UID: "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1"). InnerVolumeSpecName "kube-api-access-txm54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:48.834373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.834307    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca" pod "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" (UID: "2e34c02b-b2bf-41ef-b34b-5f24194a9ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:48.834429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.834398    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd" pod "042f074d-5258-48d8-ba14-984d23ad5965" (UID: "042f074d-5258-48d8-ba14-984d23ad5965"). InnerVolumeSpecName "kube-api-access-dtgfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:48.935058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.934959    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:48.935058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.934998    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgfd\" (UniqueName: \"kubernetes.io/projected/042f074d-5258-48d8-ba14-984d23ad5965-kube-api-access-dtgfd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:48.935058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:48.935008    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txm54\" (UniqueName: \"kubernetes.io/projected/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1-kube-api-access-txm54\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:49.482489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.482454    6738 generic.go:356] "Generic (PLEG): container finished" podID="042f074d-5258-48d8-ba14-984d23ad5965" containerID="2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7" exitCode=137
Oct 30 07:31:49.482911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.482501    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:49.482911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.482554    6738 scope.go:122] "RemoveContainer" containerID="2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7"
Oct 30 07:31:49.484177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.484155    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"
Oct 30 07:31:49.484322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.484152    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6" event={"ID":"2e34c02b-b2bf-41ef-b34b-5f24194a9ab1","Type":"ContainerDied","Data":"be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773"}
Oct 30 07:31:49.484322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.484265    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be0624d9e6d125098a692aca40050a74e0635530802626a7e4f0f6f5d1ff8773"
Oct 30 07:31:49.484515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.484460    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7436-36bdb4" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" containerName="test-curl-7436-36bdb4" containerID="cri-o://522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468" gracePeriod=2
Oct 30 07:31:49.498249 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.498214    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:49.516042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.516018    6738 scope.go:122] "RemoveContainer" containerID="2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7"
Oct 30 07:31:49.516424 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.516398    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7\": container with ID starting with 2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7 not found: ID does not exist" containerID="2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7"
Oct 30 07:31:49.516520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.516433    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7"} err="failed to get container status \"2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7\": rpc error: code = NotFound desc = could not find container \"2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7\": container with ID starting with 2336b40ced53c1fe913f8f92304106b5570aac1808381ef299a601dd1b4857f7 not found: ID does not exist"
Oct 30 07:31:49.525986 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.525938    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:49.554670 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.554635    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7433-652852\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="042f074d-5258-48d8-ba14-984d23ad5965" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:49.575923 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.568559    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7433-652852\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="042f074d-5258-48d8-ba14-984d23ad5965" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:49.582481 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.582433    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:49.658093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.658066    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:49.672059 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.672020    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:49.685702 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.685668    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7433-652852\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="042f074d-5258-48d8-ba14-984d23ad5965" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:49.693338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.693262    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00209875-65ae-4e52-8cf8-b8a93873986a" path="/var/lib/kubelet/pods/00209875-65ae-4e52-8cf8-b8a93873986a/volumes"
Oct 30 07:31:49.693834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.693812    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042f074d-5258-48d8-ba14-984d23ad5965" path="/var/lib/kubelet/pods/042f074d-5258-48d8-ba14-984d23ad5965/volumes"
Oct 30 07:31:49.705344 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.705261    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:49.718998 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:49.718957    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7433-652852\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="042f074d-5258-48d8-ba14-984d23ad5965" pod="opendatahub/test-curl-7433-652852"
Oct 30 07:31:49.745197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.745165    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\" (UniqueName: \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\") pod \"bab8cff0-6349-46bd-b975-1ccad6447144\" (UID: \"bab8cff0-6349-46bd-b975-1ccad6447144\") "
Oct 30 07:31:49.747405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.747378    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf" pod "bab8cff0-6349-46bd-b975-1ccad6447144" (UID: "bab8cff0-6349-46bd-b975-1ccad6447144"). InnerVolumeSpecName "kube-api-access-vfwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:49.847536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:49.847499    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwlf\" (UniqueName: \"kubernetes.io/projected/bab8cff0-6349-46bd-b975-1ccad6447144-kube-api-access-vfwlf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:50.226538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.226502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:50.341269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.341229    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:50.493625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.493535    6738 generic.go:356] "Generic (PLEG): container finished" podID="bab8cff0-6349-46bd-b975-1ccad6447144" containerID="7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d" exitCode=137
Oct 30 07:31:50.493625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.493593    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-826d85"
Oct 30 07:31:50.493625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.493624    6738 scope.go:122] "RemoveContainer" containerID="7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d"
Oct 30 07:31:50.503856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.503834    6738 scope.go:122] "RemoveContainer" containerID="7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d"
Oct 30 07:31:50.504197 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:50.504156    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d\": container with ID starting with 7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d not found: ID does not exist" containerID="7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d"
Oct 30 07:31:50.504247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.504205    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d"} err="failed to get container status \"7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d\": rpc error: code = NotFound desc = could not find container \"7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d\": container with ID starting with 7d138a64d02ceb0cdc9fb6c8c7cc206d9b3994ac43c1adb6ac3e4b5b28900a4d not found: ID does not exist"
Oct 30 07:31:50.736617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.736586    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 07:31:50.737406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737384    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" containerName="revoke-keys"
Oct 30 07:31:50.737406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737406    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" containerName="revoke-keys"
Oct 30 07:31:50.737591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737423    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" containerName="test-curl-7436-36bdb4"
Oct 30 07:31:50.737591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737431    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" containerName="test-curl-7436-36bdb4"
Oct 30 07:31:50.737591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737460    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" containerName="revoke-keys"
Oct 30 07:31:50.737591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737468    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" containerName="revoke-keys"
Oct 30 07:31:50.737798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737649    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" containerName="revoke-keys"
Oct 30 07:31:50.737798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737679    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" containerName="revoke-keys"
Oct 30 07:31:50.737798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.737689    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" containerName="test-curl-7436-36bdb4"
Oct 30 07:31:50.742879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.742857    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.759013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.758988    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f5081-openshift-default-dockercfg-ntqwn\""
Oct 30 07:31:50.776948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.776915    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 07:31:50.809592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.809494    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.864648    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.864709    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5dt\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.864763    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.864888    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.864973    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.865024    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.865096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.865121    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.866329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.865173    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988349    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988408    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5dt\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988462    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988535    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988629    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988679    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988761    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988785    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.988837    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.989469    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:50.993323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:50.990165    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.003330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.001651    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.003330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.002873    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.003330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.003155    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.013769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.007685    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.017734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.017390    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.032980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.032946    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.033144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.033002    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5dt\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.061486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.061452    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:51.305757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.305657    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:51.418251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.418223    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 07:31:51.418741 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:51.418718    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d78a9cd_852b_4cea_91d4_ce66d8027322.slice/crio-2a77d03bebbddef133867ff129db13bcf47374b667174ad1db9e5bdeaa7f8cb3 WatchSource:0}: Error finding container 2a77d03bebbddef133867ff129db13bcf47374b667174ad1db9e5bdeaa7f8cb3: Status 404 returned error can't find the container with id 2a77d03bebbddef133867ff129db13bcf47374b667174ad1db9e5bdeaa7f8cb3
Oct 30 07:31:51.421774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.421738    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:51.421878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.421812    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:51.421878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.421844    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:51.505787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.505752    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerStarted","Data":"2a77d03bebbddef133867ff129db13bcf47374b667174ad1db9e5bdeaa7f8cb3"}
Oct 30 07:31:51.696571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.696535    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab8cff0-6349-46bd-b975-1ccad6447144" path="/var/lib/kubelet/pods/bab8cff0-6349-46bd-b975-1ccad6447144/volumes"
Oct 30 07:31:51.706813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.706787    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:51.800681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.800649    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\" (UniqueName: \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\") pod \"b488a363-5579-46c0-af4d-e35e0fa91710\" (UID: \"b488a363-5579-46c0-af4d-e35e0fa91710\") "
Oct 30 07:31:51.802800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.802774    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns" pod "b488a363-5579-46c0-af4d-e35e0fa91710" (UID: "b488a363-5579-46c0-af4d-e35e0fa91710"). InnerVolumeSpecName "kube-api-access-zjkns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:51.902203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:51.902111    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkns\" (UniqueName: \"kubernetes.io/projected/b488a363-5579-46c0-af4d-e35e0fa91710-kube-api-access-zjkns\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:52.288128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.288041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:52.513923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.513886    6738 generic.go:356] "Generic (PLEG): container finished" podID="b488a363-5579-46c0-af4d-e35e0fa91710" containerID="522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468" exitCode=137
Oct 30 07:31:52.514394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.513929    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:52.514394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.513991    6738 scope.go:122] "RemoveContainer" containerID="522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468"
Oct 30 07:31:52.516114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.516077    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerStarted","Data":"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"}
Oct 30 07:31:52.525465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.525443    6738 scope.go:122] "RemoveContainer" containerID="522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468"
Oct 30 07:31:52.525796 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:52.525773    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468\": container with ID starting with 522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468 not found: ID does not exist" containerID="522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468"
Oct 30 07:31:52.525861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.525804    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468"} err="failed to get container status \"522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468\": rpc error: code = NotFound desc = could not find container \"522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468\": container with ID starting with 522477c7fa54fb86ea2dc391be97040de5d842a9cd5e1fe6d5ad243ce7860468 not found: ID does not exist"
Oct 30 07:31:52.587660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:52.587555    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podStartSLOduration=2.587540179 podStartE2EDuration="2.587540179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:52.585484996 +0000 UTC m=+2073.018248889" watchObservedRunningTime="2026-10-30 07:31:52.587540179 +0000 UTC m=+2073.020304159"
Oct 30 07:31:52.599354 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:52.599271    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:53.062212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.062171    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:53.068624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.068595    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:53.141736 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:53.141697    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:53.524236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.524202    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:53.525336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.525286    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:31:53.583823 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:31:53.583777    6738 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7436-36bdb4\" is forbidden: User \"system:node:ip-10-0-139-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-124.ec2.internal' and this object" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" pod="opendatahub/test-curl-7436-36bdb4"
Oct 30 07:31:53.717157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.717113    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b488a363-5579-46c0-af4d-e35e0fa91710" path="/var/lib/kubelet/pods/b488a363-5579-46c0-af4d-e35e0fa91710/volumes"
Oct 30 07:31:53.776608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.776512    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:53.947697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.947656    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"]
Oct 30 07:31:53.953848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.953814    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:53.968827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.968778    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:31:53.968995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.968958    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:31:53.970363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:53.970280    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"]
Oct 30 07:31:54.045960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.045875    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2sgd\" (UniqueName: \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.046121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.045989    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.046121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.046054    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.147839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.147802    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2sgd\" (UniqueName: \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.148043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.147887    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.148043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.147940    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.148598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.148570    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.150322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.150286    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.171934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.171891    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2sgd\" (UniqueName: \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\") pod \"maas-api-revoke-sub-07442568-dw4s2\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.269425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.269385    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:54.452958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.452915    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:54.592749 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:54.592701    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cb3feb_ee20_4ecb_8645_fbc4f6096aaf.slice/crio-effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b WatchSource:0}: Error finding container effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b: Status 404 returned error can't find the container with id effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b
Oct 30 07:31:54.593246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:54.592932    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"]
Oct 30 07:31:55.140201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.140114    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:55.261765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.261728    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:55.428014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.427926    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" podUID="48fd1a4d-fdb7-426d-8381-14ef5874473c" containerName="main" containerID="cri-o://d56df7d2e88f6a1aa01884889d47257473e3ed13eac1ba108ce2bf65e63d9d91" gracePeriod=45
Oct 30 07:31:55.562175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.562141    6738 generic.go:356] "Generic (PLEG): container finished" podID="39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" containerID="1bc3987625afba7f2b512d4f9e0419e2ff7ae0f0948f221ab515ec419dda1de1" exitCode=0
Oct 30 07:31:55.562400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.562186    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2" event={"ID":"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf","Type":"ContainerDied","Data":"1bc3987625afba7f2b512d4f9e0419e2ff7ae0f0948f221ab515ec419dda1de1"}
Oct 30 07:31:55.562400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.562256    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2" event={"ID":"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf","Type":"ContainerStarted","Data":"effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b"}
Oct 30 07:31:55.564119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.564092    6738 generic.go:356] "Generic (PLEG): container finished" podID="48fd1a4d-fdb7-426d-8381-14ef5874473c" containerID="d56df7d2e88f6a1aa01884889d47257473e3ed13eac1ba108ce2bf65e63d9d91" exitCode=0
Oct 30 07:31:55.564237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.564164    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" event={"ID":"48fd1a4d-fdb7-426d-8381-14ef5874473c","Type":"ContainerDied","Data":"d56df7d2e88f6a1aa01884889d47257473e3ed13eac1ba108ce2bf65e63d9d91"}
Oct 30 07:31:55.734661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.734629    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:31:55.769132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769092    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769176    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\" (UniqueName: \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769278    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769397    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\" (UniqueName: \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769415    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:55.769452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769437    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769475    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\") pod \"48fd1a4d-fdb7-426d-8381-14ef5874473c\" (UID: \"48fd1a4d-fdb7-426d-8381-14ef5874473c\") "
Oct 30 07:31:55.769627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769584    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:55.769801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.769776    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:55.770214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.770189    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:55.770413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.770394    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:55.770519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.770505    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:55.771935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.771910    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "kube-api-access-q6kqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:55.772055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.772017    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:55.772713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.772696    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm" pod "48fd1a4d-fdb7-426d-8381-14ef5874473c" (UID: "48fd1a4d-fdb7-426d-8381-14ef5874473c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:55.871958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.871924    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6kqg\" (UniqueName: \"kubernetes.io/projected/48fd1a4d-fdb7-426d-8381-14ef5874473c-kube-api-access-q6kqg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:55.871958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.871956    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fd1a4d-fdb7-426d-8381-14ef5874473c-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:55.871958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:55.871968    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fd1a4d-fdb7-426d-8381-14ef5874473c-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:56.128076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.128046    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"]
Oct 30 07:31:56.128745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.128721    6738 cpu_manager.go:414] "containerMap: removing container" podUID="48fd1a4d-fdb7-426d-8381-14ef5874473c" containerName="main"
Oct 30 07:31:56.128792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.128747    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fd1a4d-fdb7-426d-8381-14ef5874473c" containerName="main"
Oct 30 07:31:56.128891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.128881    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fd1a4d-fdb7-426d-8381-14ef5874473c" containerName="main"
Oct 30 07:31:56.132808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.132785    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.150114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.150082    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"]
Oct 30 07:31:56.176142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.176098    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.176391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.176318    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.176475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.176395    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrcr\" (UniqueName: \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.277768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.277732    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.277963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.277880    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.278071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.278048    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrcr\" (UniqueName: \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.278800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.278768    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.280440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.280413    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.280775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.280743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:56.301930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.301896    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrcr\" (UniqueName: \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\") pod \"maas-api-revoke-sub-02708227-mt22m\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.449350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.449257    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:56.525345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.522267    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:56.608328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.607262    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"
Oct 30 07:31:56.616815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.614707    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4" event={"ID":"48fd1a4d-fdb7-426d-8381-14ef5874473c","Type":"ContainerDied","Data":"4e289adc1d0d1a99892c79b96ea9c35c96ca1e9c0ca4970f6dae38448b2a1ed4"}
Oct 30 07:31:56.616815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.614776    6738 scope.go:122] "RemoveContainer" containerID="d56df7d2e88f6a1aa01884889d47257473e3ed13eac1ba108ce2bf65e63d9d91"
Oct 30 07:31:56.675209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.675162    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:31:56.681368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.681317    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/auth-test-model-f5ceca-kserve-56d89f569-9fpj4"]
Oct 30 07:31:56.761836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.761807    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"]
Oct 30 07:31:56.762210 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:56.762184    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be02b41_8bf8_4fb4_b0a2_f0e9fca269c0.slice/crio-c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190 WatchSource:0}: Error finding container c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190: Status 404 returned error can't find the container with id c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190
Oct 30 07:31:56.803721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.801697    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:31:56.809260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.809238    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.826726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.826514    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a8f1df-openshift-default-dockercfg-9q6vn\""
Oct 30 07:31:56.827075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.826913    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:31:56.830531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.830505    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:56.860485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.860446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:56.898968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.898926    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899016    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899074    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89fp\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899110    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899159    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899208    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899378    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.899474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.899454    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:56.999638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:56.999601    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:57.000712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.000684    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\" (UniqueName: \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\") pod \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") "
Oct 30 07:31:57.000820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.000779    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\") pod \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") "
Oct 30 07:31:57.001039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001020    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\") pod \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\" (UID: \"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf\") "
Oct 30 07:31:57.001325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001270    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001373    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca" pod "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" (UID: "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:57.001436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001384    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001449    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001539    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89fp\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001599    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001619    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001626    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001657    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001738    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001788    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.001905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.001868    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:57.002185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.002161    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.002567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.002544    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.002691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.002670    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.002787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.002724    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.003576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.003548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd" pod "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" (UID: "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf"). InnerVolumeSpecName "kube-api-access-v2sgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:57.004629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.004608    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca" pod "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" (UID: "39cb3feb-ee20-4ecb-8645-fbc4f6096aaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:57.004807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.004787    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.005966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.005945    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.024944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.024914    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.025192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.025165    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89fp\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\") pod \"e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") " pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.103223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.103187    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2sgd\" (UniqueName: \"kubernetes.io/projected/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-kube-api-access-v2sgd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:57.103223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.103226    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:57.120530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.120481    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:57.144418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.144359    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:57.214875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.214842    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 07:31:57.219057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.219033    6738 cpu_manager.go:414] "containerMap: removing container" podUID="39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" containerName="revoke-keys"
Oct 30 07:31:57.219152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.219061    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" containerName="revoke-keys"
Oct 30 07:31:57.219387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.219307    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" containerName="revoke-keys"
Oct 30 07:31:57.225150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.225124    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.241367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.241285    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 07:31:57.241526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.241423    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f5081\""
Oct 30 07:31:57.241526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.241515    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f5081-dockercfg-8vc5s\""
Oct 30 07:31:57.353320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.326344    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.353320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.326483    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc4c\" (UniqueName: \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.421318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.421170    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 07:31:57.443583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.439099    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.443583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.442059    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.443583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.442198    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc4c\" (UniqueName: \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.443890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.443751    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.493393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.492669    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 07:31:57.527232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.524429    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:31:57.527471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.527435    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc4c\" (UniqueName: \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\") pod \"payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.538267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.537938    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:57.538267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.538028    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:57.538267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.538075    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:31:57.544380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.544345    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.544558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.544538    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ncx\" (UniqueName: \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.549052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.549023    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:57.616219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.616171    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerStarted","Data":"addfb5c12bf0967602bfbc82d69c5d9303060b1796b5e61181013cff82f329db"}
Oct 30 07:31:57.618105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.618072    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m" event={"ID":"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0","Type":"ContainerStarted","Data":"6000344ea2d67552d270b46110798ddc6e17d750dcdd4aa2fda17cce10ad9df5"}
Oct 30 07:31:57.618259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.618138    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m" event={"ID":"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0","Type":"ContainerStarted","Data":"c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190"}
Oct 30 07:31:57.619897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.619873    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"
Oct 30 07:31:57.620051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.619871    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2" event={"ID":"39cb3feb-ee20-4ecb-8645-fbc4f6096aaf","Type":"ContainerDied","Data":"effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b"}
Oct 30 07:31:57.620051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.619999    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="effae4c7199f1cc482e74d27770fd950a827619eea740c8b9403ab30731a8e1b"
Oct 30 07:31:57.646208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.646172    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.646404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.646377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ncx\" (UniqueName: \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.647085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.647063    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.661259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.661210    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m" podStartSLOduration=1.66119631 podStartE2EDuration="1.66119631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:57.658956187 +0000 UTC m=+2078.091720082" watchObservedRunningTime="2026-10-30 07:31:57.66119631 +0000 UTC m=+2078.093960203"
Oct 30 07:31:57.672394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.672368    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ncx\" (UniqueName: \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\") pod \"payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.703828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.703792    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fd1a4d-fdb7-426d-8381-14ef5874473c" path="/var/lib/kubelet/pods/48fd1a4d-fdb7-426d-8381-14ef5874473c/volumes"
Oct 30 07:31:57.859752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.859535    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:57.934467 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:57.934393    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4295b2e2_3177_4e60_ba21_eb456273e819.slice/crio-a34883babbeb8c92313972d5ed11617809d139b910b06fe61448e260ffcce986 WatchSource:0}: Error finding container a34883babbeb8c92313972d5ed11617809d139b910b06fe61448e260ffcce986: Status 404 returned error can't find the container with id a34883babbeb8c92313972d5ed11617809d139b910b06fe61448e260ffcce986
Oct 30 07:31:57.943563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:57.941843    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 07:31:58.175255 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:31:58.175220    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a262e8_dbc6_4082_bf1b_3dc1655d8ebd.slice/crio-a5ad6bb671db8ecb64183757999f5700a8fefb0977ead2f8d4b00560d1e17bb5 WatchSource:0}: Error finding container a5ad6bb671db8ecb64183757999f5700a8fefb0977ead2f8d4b00560d1e17bb5: Status 404 returned error can't find the container with id a5ad6bb671db8ecb64183757999f5700a8fefb0977ead2f8d4b00560d1e17bb5
Oct 30 07:31:58.176176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.176084    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 07:31:58.539934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.539841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:31:58.650498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.650238    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" event={"ID":"4295b2e2-3177-4e60-ba21-eb456273e819","Type":"ContainerStarted","Data":"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"}
Oct 30 07:31:58.650498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.650363    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" event={"ID":"4295b2e2-3177-4e60-ba21-eb456273e819","Type":"ContainerStarted","Data":"a34883babbeb8c92313972d5ed11617809d139b910b06fe61448e260ffcce986"}
Oct 30 07:31:58.652016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.651248    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:31:58.653691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.653660    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" event={"ID":"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd","Type":"ContainerStarted","Data":"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"}
Oct 30 07:31:58.653853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.653715    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" event={"ID":"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd","Type":"ContainerStarted","Data":"a5ad6bb671db8ecb64183757999f5700a8fefb0977ead2f8d4b00560d1e17bb5"}
Oct 30 07:31:58.653853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.653757    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:31:58.656007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.655977    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerStarted","Data":"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"}
Oct 30 07:31:58.657850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.657821    6738 generic.go:356] "Generic (PLEG): container finished" podID="0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" containerID="6000344ea2d67552d270b46110798ddc6e17d750dcdd4aa2fda17cce10ad9df5" exitCode=0
Oct 30 07:31:58.657957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.657852    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m" event={"ID":"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0","Type":"ContainerDied","Data":"6000344ea2d67552d270b46110798ddc6e17d750dcdd4aa2fda17cce10ad9df5"}
Oct 30 07:31:58.697076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.696962    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" podStartSLOduration=1.696948721 podStartE2EDuration="1.696948721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:58.693498775 +0000 UTC m=+2079.126262668" watchObservedRunningTime="2026-10-30 07:31:58.696948721 +0000 UTC m=+2079.129712613"
Oct 30 07:31:58.740160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.740100    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podStartSLOduration=2.740081984 podStartE2EDuration="2.740081984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:58.739169964 +0000 UTC m=+2079.171933852" watchObservedRunningTime="2026-10-30 07:31:58.740081984 +0000 UTC m=+2079.172845878"
Oct 30 07:31:58.781139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:58.781079    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" podStartSLOduration=1.7810589559999999 podStartE2EDuration="1.781058956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:58.77988882 +0000 UTC m=+2079.212652716" watchObservedRunningTime="2026-10-30 07:31:58.781058956 +0000 UTC m=+2079.213822850"
Oct 30 07:31:59.145133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.145049    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:59.151519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.151489    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:59.666013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.665668    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:59.666394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.666372    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:31:59.898278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.898248    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:31:59.984137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.983995    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\" (UniqueName: \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\") pod \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") "
Oct 30 07:31:59.984137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.984044    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\") pod \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") "
Oct 30 07:31:59.984459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.984178    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\") pod \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\" (UID: \"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0\") "
Oct 30 07:31:59.984533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.984502    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca" pod "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" (UID: "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:59.984923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.984896    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:31:59.986403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.986377    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca" pod "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" (UID: "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:59.986529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:31:59.986436    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr" pod "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" (UID: "0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0"). InnerVolumeSpecName "kube-api-access-qbrcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:00.086822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.086790    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrcr\" (UniqueName: \"kubernetes.io/projected/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-kube-api-access-qbrcr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:00.086822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.086820    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:00.529482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.529449    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:00.673896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.673861    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"
Oct 30 07:32:00.673896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.673892    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m" event={"ID":"0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0","Type":"ContainerDied","Data":"c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190"}
Oct 30 07:32:00.674098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:00.673974    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c169e69f0dd25777dab4829c5ae6449b91d00cc04c38d50007cd7fae0ce53190"
Oct 30 07:32:01.272188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:01.272135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:01.400627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:01.400592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:02.534937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:02.534898    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:03.033040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.032985    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:03.145385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.145349    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:03.205309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.205264    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:32:03.206016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.206000    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" containerName="revoke-keys"
Oct 30 07:32:03.206055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.206018    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" containerName="revoke-keys"
Oct 30 07:32:03.206172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.206159    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" containerName="revoke-keys"
Oct 30 07:32:03.210126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.210103    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.227855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.227821    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:32:03.240355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.235483    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a8f1df-dockercfg-tpt5g\""
Oct 30 07:32:03.240355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.236006    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a8f1df\""
Oct 30 07:32:03.379467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.372098    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q49c\" (UniqueName: \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.379467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.372211    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.502313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.487451    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.502313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.487859    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q49c\" (UniqueName: \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.502313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.500148    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.532316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.531864    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:32:03.555380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.555345    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q49c\" (UniqueName: \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\") pod \"payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.557814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.557787    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.561731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.561700    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:32:03.695242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.695141    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd25f\" (UniqueName: \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.695242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.695205    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.796962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.796918    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd25f\" (UniqueName: \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.797175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.797062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.797829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.797807    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.820583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.820552    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd25f\" (UniqueName: \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\") pod \"payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:03.826427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.826400    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:03.887796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:03.887761    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:04.133012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.132981    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:32:04.133160 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:04.133136    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7efff85_fd9e_4e94_a124_8f535c889d80.slice/crio-574a44b74b95e62f8a72d060c6f34dd28e689fa748f6fcc71cb2fbce29f5b874 WatchSource:0}: Error finding container 574a44b74b95e62f8a72d060c6f34dd28e689fa748f6fcc71cb2fbce29f5b874: Status 404 returned error can't find the container with id 574a44b74b95e62f8a72d060c6f34dd28e689fa748f6fcc71cb2fbce29f5b874
Oct 30 07:32:04.217940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.217913    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:32:04.218048 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:04.218025    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b48ee79_9976_42f2_8c53_5a13f51a056f.slice/crio-90fe6f16e29173f850784dad7a4e49d8d25a5a01879cb336803639fe86827be7 WatchSource:0}: Error finding container 90fe6f16e29173f850784dad7a4e49d8d25a5a01879cb336803639fe86827be7: Status 404 returned error can't find the container with id 90fe6f16e29173f850784dad7a4e49d8d25a5a01879cb336803639fe86827be7
Oct 30 07:32:04.459498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.459457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:04.705742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.705710    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" event={"ID":"4b48ee79-9976-42f2-8c53-5a13f51a056f","Type":"ContainerStarted","Data":"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"}
Oct 30 07:32:04.706233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.705769    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" event={"ID":"4b48ee79-9976-42f2-8c53-5a13f51a056f","Type":"ContainerStarted","Data":"90fe6f16e29173f850784dad7a4e49d8d25a5a01879cb336803639fe86827be7"}
Oct 30 07:32:04.706233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.705798    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:04.707851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.707815    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" event={"ID":"f7efff85-fd9e-4e94-a124-8f535c889d80","Type":"ContainerStarted","Data":"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"}
Oct 30 07:32:04.707971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.707879    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" event={"ID":"f7efff85-fd9e-4e94-a124-8f535c889d80","Type":"ContainerStarted","Data":"574a44b74b95e62f8a72d060c6f34dd28e689fa748f6fcc71cb2fbce29f5b874"}
Oct 30 07:32:04.707971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.707907    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:04.752527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.752470    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" podStartSLOduration=1.752455857 podStartE2EDuration="1.752455857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:04.74898577 +0000 UTC m=+2085.181749664" watchObservedRunningTime="2026-10-30 07:32:04.752455857 +0000 UTC m=+2085.185219749"
Oct 30 07:32:04.790982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:04.790926    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" podStartSLOduration=1.790912686 podStartE2EDuration="1.790912686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:04.790347333 +0000 UTC m=+2085.223111220" watchObservedRunningTime="2026-10-30 07:32:04.790912686 +0000 UTC m=+2085.223676579"
Oct 30 07:32:06.336287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:06.336255    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:06.475752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:06.473278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:07.520482 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:07.520441    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:32:08.330805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:08.330766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:09.668762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:09.668725    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 07:32:09.669182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:09.668878    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 07:32:10.202038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:10.202001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:12.011709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:12.011668    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:13.869204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:13.869161    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:14.393571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:14.393533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:15.720272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:15.720240    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:32:15.720746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:15.720356    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:32:16.043440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.043340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:16.173454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.173417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:16.459238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.459197    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 07:32:16.464153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.464129    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.486116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.486073    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8e7948-openshift-default-dockercfg-zbjgh\""
Oct 30 07:32:16.524377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.524342    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 07:32:16.544958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.544923    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.544994    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545083    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8nm\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545177    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545227    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545355    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545712    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545776    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.545889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.545839    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.631317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.629006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648615    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648716    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648774    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8nm\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648827    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648879    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.648979    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.649145    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.649217    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.649301    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.649999    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.650344    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.650616    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.651842    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.652320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.652197    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.672315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.668351    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.687540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.687507    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.711446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.711317    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8nm\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.726131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.725684    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\") pod \"e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") " pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.785137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.785106    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:16.838420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:16.838378    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:17.167943 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:17.167915    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bf58b8_3623_494a_bb85_1b013279b919.slice/crio-91e7a128d6b9f0f0975195f6bb142f88775c9ab826882efe76db68d8174e5c88 WatchSource:0}: Error finding container 91e7a128d6b9f0f0975195f6bb142f88775c9ab826882efe76db68d8174e5c88: Status 404 returned error can't find the container with id 91e7a128d6b9f0f0975195f6bb142f88775c9ab826882efe76db68d8174e5c88
Oct 30 07:32:17.168851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.168825    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 07:32:17.171652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.171610    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:32:17.171749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.171703    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:32:17.171798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.171773    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:32:17.898794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.898758    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerStarted","Data":"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"}
Oct 30 07:32:17.899340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.898825    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerStarted","Data":"91e7a128d6b9f0f0975195f6bb142f88775c9ab826882efe76db68d8174e5c88"}
Oct 30 07:32:17.948796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:17.948725    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podStartSLOduration=1.948709135 podStartE2EDuration="1.948709135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:17.945159153 +0000 UTC m=+2098.377923046" watchObservedRunningTime="2026-10-30 07:32:17.948709135 +0000 UTC m=+2098.381473028"
Oct 30 07:32:18.052230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.052194    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"]
Oct 30 07:32:18.070321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.058871    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.083587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.083511    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:32:18.083913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.083895    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:32:18.129415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.129376    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"]
Oct 30 07:32:18.208357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.208257    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.208542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.208433    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7skg\" (UniqueName: \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.208542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.208481    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.274185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.274155    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:18.309556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.309522    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7skg\" (UniqueName: \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.309780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.309598    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.309842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.309787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.311897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.310848    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.392418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.392276    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7skg\" (UniqueName: \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.407371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.405993    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fde6cc8-wklrq\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.420787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.420744    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:18.730333 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:18.730261    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947cebf4_ad18_4c5c_8f25_6bfbc4e9c76c.slice/crio-658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de WatchSource:0}: Error finding container 658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de: Status 404 returned error can't find the container with id 658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de
Oct 30 07:32:18.730333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.730305    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"]
Oct 30 07:32:18.786256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.786224    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:18.792200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.792173    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:18.910156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.910121    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq" event={"ID":"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c","Type":"ContainerStarted","Data":"658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de"}
Oct 30 07:32:18.910606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.910507    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:18.911674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:18.911653    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:32:19.919433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:19.919399    6738 generic.go:356] "Generic (PLEG): container finished" podID="947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" containerID="25f7f7a3b864a0c9a8ab7157a3986489b901244b1d11806a81c7fbb47bd4ce3b" exitCode=0
Oct 30 07:32:19.919871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:19.919483    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq" event={"ID":"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c","Type":"ContainerDied","Data":"25f7f7a3b864a0c9a8ab7157a3986489b901244b1d11806a81c7fbb47bd4ce3b"}
Oct 30 07:32:20.387919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:20.387880    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:20.525525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:20.525487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:20.648369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:20.647287    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:21.106711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.106684    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:21.159321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.159273    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\") pod \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") "
Oct 30 07:32:21.159522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.159397    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\") pod \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") "
Oct 30 07:32:21.159522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.159420    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\" (UniqueName: \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\") pod \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\" (UID: \"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c\") "
Oct 30 07:32:21.159820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.159793    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca" pod "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" (UID: "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:32:21.160241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.160212    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.161902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.161862    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg" pod "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" (UID: "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c"). InnerVolumeSpecName "kube-api-access-m7skg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:21.162227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.162198    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca" pod "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" (UID: "947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:32:21.261835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.261750    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7skg\" (UniqueName: \"kubernetes.io/projected/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-kube-api-access-m7skg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.261835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.261781    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.937888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.937855    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"
Oct 30 07:32:21.937888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.937866    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq" event={"ID":"947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c","Type":"ContainerDied","Data":"658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de"}
Oct 30 07:32:21.938100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:21.937917    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="658d685416c61186ab0a4df2e0a4781c7764103fe9ecdd9cc8810d79d30755de"
Oct 30 07:32:22.143070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.143029    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:22.265668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.265631    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:22.328478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.328445    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 07:32:22.329165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.329151    6738 cpu_manager.go:414] "containerMap: removing container" podUID="947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" containerName="revoke-keys"
Oct 30 07:32:22.329223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.329167    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" containerName="revoke-keys"
Oct 30 07:32:22.329327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.329316    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" containerName="revoke-keys"
Oct 30 07:32:22.333017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.332997    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.349443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.348912    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8e7948-dockercfg-65k9x\""
Oct 30 07:32:22.349443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.349121    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8e7948\""
Oct 30 07:32:22.354530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.354501    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 07:32:22.496367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.494348    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsbp\" (UniqueName: \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.496367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.494527    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.596244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.596154    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.602364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.601025    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsbp\" (UniqueName: \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.602364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.602254    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.639200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.639160    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:22.675387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.670765    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsbp\" (UniqueName: \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\") pod \"payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:22.947959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:22.947929    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:23.258927 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:23.258895    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ae4571_bd10_4a89_860f_65de96573cf6.slice/crio-7132eba7b6d0a5c586a7efb5445df62ff5c272b94efa03bed408a6ecfc2cd29e WatchSource:0}: Error finding container 7132eba7b6d0a5c586a7efb5445df62ff5c272b94efa03bed408a6ecfc2cd29e: Status 404 returned error can't find the container with id 7132eba7b6d0a5c586a7efb5445df62ff5c272b94efa03bed408a6ecfc2cd29e
Oct 30 07:32:23.259403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:23.258938    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 07:32:23.954980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:23.954936    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" event={"ID":"19ae4571-bd10-4a89-860f-65de96573cf6","Type":"ContainerStarted","Data":"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"}
Oct 30 07:32:23.955200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:23.955017    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" event={"ID":"19ae4571-bd10-4a89-860f-65de96573cf6","Type":"ContainerStarted","Data":"7132eba7b6d0a5c586a7efb5445df62ff5c272b94efa03bed408a6ecfc2cd29e"}
Oct 30 07:32:23.955325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:23.955266    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:24.001280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:24.001218    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" podStartSLOduration=2.001202938 podStartE2EDuration="2.001202938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:23.997151357 +0000 UTC m=+2104.429915252" watchObservedRunningTime="2026-10-30 07:32:24.001202938 +0000 UTC m=+2104.433966831"
Oct 30 07:32:24.527637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:24.527593    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:26.322454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.322419    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 07:32:26.327522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.327498    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.348551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.348519    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 07:32:26.366343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.360636    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0f5081-serving-cert\""
Oct 30 07:32:26.430763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.430717    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:26.510175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.510139    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.510358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.510327    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.510490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.510473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqx5\" (UniqueName: \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.510587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.510573    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.546574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.546535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:26.614373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.613639    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqx5\" (UniqueName: \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.614373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.613767    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.614373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.613864    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.614373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.613989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.618394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.616340    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.671321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.665087    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.677316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.676056    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqx5\" (UniqueName: \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.677316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.676633    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:26.943722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:26.943679    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:27.258793 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:27.258760    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5938677c_2476_4e6f_a1f3_3192c897a5eb.slice/crio-794c852b200fa8f10f9ef0799332849b87b36636b993e2d0c8d35f0604acafaf WatchSource:0}: Error finding container 794c852b200fa8f10f9ef0799332849b87b36636b993e2d0c8d35f0604acafaf: Status 404 returned error can't find the container with id 794c852b200fa8f10f9ef0799332849b87b36636b993e2d0c8d35f0604acafaf
Oct 30 07:32:27.259815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:27.259789    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 07:32:27.995717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:27.995681    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" event={"ID":"5938677c-2476-4e6f-a1f3-3192c897a5eb","Type":"ContainerStarted","Data":"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"}
Oct 30 07:32:27.996155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:27.995740    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" event={"ID":"5938677c-2476-4e6f-a1f3-3192c897a5eb","Type":"ContainerStarted","Data":"794c852b200fa8f10f9ef0799332849b87b36636b993e2d0c8d35f0604acafaf"}
Oct 30 07:32:27.996155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:27.995771    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:28.040462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.040406    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" podStartSLOduration=1.4603181250000001 podStartE2EDuration="2.040391096s" totalImagesPullingTime="580.072971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:28.038012148 +0000 UTC m=+2108.470776032" watchObservedRunningTime="2026-10-30 07:32:28.040391096 +0000 UTC m=+2108.473154988"
Oct 30 07:32:28.444513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.444459    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"]
Oct 30 07:32:28.449205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.449178    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.464509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.464466    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:32:28.464729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.464519    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:32:28.468102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.468077    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"]
Oct 30 07:32:28.585132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.585091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:28.647072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.638606    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.647072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.638703    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjdj\" (UniqueName: \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.647072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.638744    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.740569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.740481    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.740569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.740565    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjdj\" (UniqueName: \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.740832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.740592    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.741401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.741352    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.742978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.742956    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:28.764478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:28.764445    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjdj\" (UniqueName: \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\") pod \"maas-api-revoke-sub-5a1e9fa6-5p265\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:29.064355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:29.064244    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:29.374945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:29.374915    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"]
Oct 30 07:32:29.375576 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:29.375548    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d95a29_96fb_4483_aba7_d7cab27d52ba.slice/crio-a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322 WatchSource:0}: Error finding container a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322: Status 404 returned error can't find the container with id a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322
Oct 30 07:32:30.012350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:30.012304    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265" event={"ID":"93d95a29-96fb-4483-aba7-d7cab27d52ba","Type":"ContainerStarted","Data":"fccbcc4c1b2eb7bc9d1f15c607a087b7b00bf548ea8c05474ed83b7d2e80cecf"}
Oct 30 07:32:30.012538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:30.012369    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265" event={"ID":"93d95a29-96fb-4483-aba7-d7cab27d52ba","Type":"ContainerStarted","Data":"a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322"}
Oct 30 07:32:30.056362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:30.056202    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265" podStartSLOduration=2.056166913 podStartE2EDuration="2.056166913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:30.055333483 +0000 UTC m=+2110.488097377" watchObservedRunningTime="2026-10-30 07:32:30.056166913 +0000 UTC m=+2110.488930807"
Oct 30 07:32:30.456503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:30.456468    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:31.021078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:31.021035    6738 generic.go:356] "Generic (PLEG): container finished" podID="93d95a29-96fb-4483-aba7-d7cab27d52ba" containerID="fccbcc4c1b2eb7bc9d1f15c607a087b7b00bf548ea8c05474ed83b7d2e80cecf" exitCode=0
Oct 30 07:32:31.021265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:31.021096    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265" event={"ID":"93d95a29-96fb-4483-aba7-d7cab27d52ba","Type":"ContainerDied","Data":"fccbcc4c1b2eb7bc9d1f15c607a087b7b00bf548ea8c05474ed83b7d2e80cecf"}
Oct 30 07:32:31.842986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:31.842948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:32.189571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.189532    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:32.241843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.241799    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:32.319842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.319799    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\") pod \"93d95a29-96fb-4483-aba7-d7cab27d52ba\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") "
Oct 30 07:32:32.320121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.320102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\" (UniqueName: \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\") pod \"93d95a29-96fb-4483-aba7-d7cab27d52ba\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") "
Oct 30 07:32:32.320656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.320624    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\") pod \"93d95a29-96fb-4483-aba7-d7cab27d52ba\" (UID: \"93d95a29-96fb-4483-aba7-d7cab27d52ba\") "
Oct 30 07:32:32.322286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.321237    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca" pod "93d95a29-96fb-4483-aba7-d7cab27d52ba" (UID: "93d95a29-96fb-4483-aba7-d7cab27d52ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:32:32.323994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.323965    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca" pod "93d95a29-96fb-4483-aba7-d7cab27d52ba" (UID: "93d95a29-96fb-4483-aba7-d7cab27d52ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:32:32.327714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.327683    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj" pod "93d95a29-96fb-4483-aba7-d7cab27d52ba" (UID: "93d95a29-96fb-4483-aba7-d7cab27d52ba"). InnerVolumeSpecName "kube-api-access-wkjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:32.423932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.423830    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:32.423932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.423875    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d95a29-96fb-4483-aba7-d7cab27d52ba-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:32.423932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:32.423891    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkjdj\" (UniqueName: \"kubernetes.io/projected/93d95a29-96fb-4483-aba7-d7cab27d52ba-kube-api-access-wkjdj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:33.165371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:33.145264    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265" event={"ID":"93d95a29-96fb-4483-aba7-d7cab27d52ba","Type":"ContainerDied","Data":"a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322"}
Oct 30 07:32:33.165371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:33.145347    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b2117c56db3d71d79659de46a9fc37bea319ca6f94b80d0eb96d23c6e3b322"
Oct 30 07:32:33.165371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:33.145419    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"
Oct 30 07:32:34.009557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:34.009522    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 07:32:34.415642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:34.415603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:34.545376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:34.545338    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:34.809597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:34.809503    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:34.967358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:34.967240    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 07:32:36.365006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:36.364972    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:36.500703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:36.500669    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:37.519641 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:37.519596    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:32:37.915956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:37.915920    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:38.357000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:38.356902    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:38.478380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:38.478336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:38.762369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:38.758186    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:40.302489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:40.302452    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:41.940792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:41.940753    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:42.023165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:42.023113    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:32:42.023413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:42.023384    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" containerID="cri-o://726d974006f7ca47d44c129e9d62979d0c098975be8d7bf3d9107b859bc804a3" gracePeriod=30
Oct 30 07:32:42.083932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:42.083883    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:42.403943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:42.403907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:43.651964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.651926    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:43.770306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.770250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:43.827410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.827378    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 07:32:43.828142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.828126    6738 cpu_manager.go:414] "containerMap: removing container" podUID="93d95a29-96fb-4483-aba7-d7cab27d52ba" containerName="revoke-keys"
Oct 30 07:32:43.828205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.828143    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d95a29-96fb-4483-aba7-d7cab27d52ba" containerName="revoke-keys"
Oct 30 07:32:43.828288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.828278    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d95a29-96fb-4483-aba7-d7cab27d52ba" containerName="revoke-keys"
Oct 30 07:32:43.831918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.831896    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:43.849734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.849702    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 07:32:43.856816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.856781    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:32:43.857053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.856960    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9bec5c-dockercfg-46vcs\""
Oct 30 07:32:44.019145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:43.977315    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvjb\" (UniqueName: \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.019693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.019663    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.087598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.087558    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 07:32:44.103105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.103056    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.111083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.111050    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 07:32:44.150500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.150461    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.150802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.150770    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvjb\" (UniqueName: \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.151571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.151548    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.176505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.176473    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvjb\" (UniqueName: \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\") pod \"payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.252115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.252076    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25w5\" (UniqueName: \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.252348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.252250    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.353912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.353825    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25w5\" (UniqueName: \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.354079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.353946    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.354688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.354668    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.377823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.377769    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25w5\" (UniqueName: \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\") pod \"payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.450575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.450541    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:44.465023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.464987    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:44.654035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.653997    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:44.791385 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:44.791353    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9e53ef_ecb3_4f28_ba98_8c9b61bd64b4.slice/crio-dd8e053f3dfb4cbbdcc31078ea3c08ee5ea36266d2f579c59e8a8b0ff2b42f73 WatchSource:0}: Error finding container dd8e053f3dfb4cbbdcc31078ea3c08ee5ea36266d2f579c59e8a8b0ff2b42f73: Status 404 returned error can't find the container with id dd8e053f3dfb4cbbdcc31078ea3c08ee5ea36266d2f579c59e8a8b0ff2b42f73
Oct 30 07:32:44.792382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.792355    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 07:32:44.800351 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:32:44.800319    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2e9c97_5e4f_4081_ab46_537d26cf5554.slice/crio-18a3ebbaf2f018e785344579511f0d1f95ef6e38c043c54eb2633de42f632582 WatchSource:0}: Error finding container 18a3ebbaf2f018e785344579511f0d1f95ef6e38c043c54eb2633de42f632582: Status 404 returned error can't find the container with id 18a3ebbaf2f018e785344579511f0d1f95ef6e38c043c54eb2633de42f632582
Oct 30 07:32:44.801008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:44.800984    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 07:32:45.258257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.258212    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" event={"ID":"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4","Type":"ContainerStarted","Data":"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"}
Oct 30 07:32:45.258617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.258356    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:45.258617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.258393    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" event={"ID":"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4","Type":"ContainerStarted","Data":"dd8e053f3dfb4cbbdcc31078ea3c08ee5ea36266d2f579c59e8a8b0ff2b42f73"}
Oct 30 07:32:45.260055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.260031    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" event={"ID":"ef2e9c97-5e4f-4081-ab46-537d26cf5554","Type":"ContainerStarted","Data":"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"}
Oct 30 07:32:45.260190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.260075    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" event={"ID":"ef2e9c97-5e4f-4081-ab46-537d26cf5554","Type":"ContainerStarted","Data":"18a3ebbaf2f018e785344579511f0d1f95ef6e38c043c54eb2633de42f632582"}
Oct 30 07:32:45.260190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.260132    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:45.299991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.299932    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" podStartSLOduration=2.2999148480000002 podStartE2EDuration="2.299914848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:45.299234154 +0000 UTC m=+2125.731998048" watchObservedRunningTime="2026-10-30 07:32:45.299914848 +0000 UTC m=+2125.732678743"
Oct 30 07:32:45.341591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:45.341521    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" podStartSLOduration=2.3415008840000002 podStartE2EDuration="2.341500884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:45.340025379 +0000 UTC m=+2125.772789271" watchObservedRunningTime="2026-10-30 07:32:45.341500884 +0000 UTC m=+2125.774264777"
Oct 30 07:32:46.461392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:46.461353    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kwlpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:32:46.461799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:46.461416    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:32:46.631434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:46.631386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:47.254213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.254169    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:32:47.254891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.254858    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy" containerID="cri-o://200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0" gracePeriod=30
Oct 30 07:32:47.266596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.266560    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:32:47.267853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.267815    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy" containerID="cri-o://eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d" gracePeriod=30
Oct 30 07:32:47.280214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.280184    6738 generic.go:356] "Generic (PLEG): container finished" podID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerID="726d974006f7ca47d44c129e9d62979d0c098975be8d7bf3d9107b859bc804a3" exitCode=0
Oct 30 07:32:47.280419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.280263    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" event={"ID":"b52155c4-081e-4709-b6a6-6b31eeb6c7d7","Type":"ContainerDied","Data":"726d974006f7ca47d44c129e9d62979d0c098975be8d7bf3d9107b859bc804a3"}
Oct 30 07:32:47.330399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.330375    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:32:47.418452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418416    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418459    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418542    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418642    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418757    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418823    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418856    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418757    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:47.418969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418918    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.418969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418954    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\") pod \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\" (UID: \"b52155c4-081e-4709-b6a6-6b31eeb6c7d7\") "
Oct 30 07:32:47.419114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.418972    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:47.419114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419064    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:47.419353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419239    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:47.419792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419768    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.419946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419922    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.419946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419946    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.420146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.419960    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.420146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.420121    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:32:47.421492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.421383    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:47.421492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.421406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:47.421492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.421417    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:32:47.421881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.421855    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf" pod "b52155c4-081e-4709-b6a6-6b31eeb6c7d7" (UID: "b52155c4-081e-4709-b6a6-6b31eeb6c7d7"). InnerVolumeSpecName "kube-api-access-rx2qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:47.521938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.521903    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.521938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.521935    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.522451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.521950    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2qf\" (UniqueName: \"kubernetes.io/projected/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-kube-api-access-rx2qf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.522451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.521960    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:47.522451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:47.521973    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52155c4-081e-4709-b6a6-6b31eeb6c7d7-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:48.291171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.291137    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"
Oct 30 07:32:48.291443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.291137    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn" event={"ID":"b52155c4-081e-4709-b6a6-6b31eeb6c7d7","Type":"ContainerDied","Data":"5adf60695d3f6ecdfd1cb7fd489b4aaa5664c25baccf30d45c0e3ebe3cd07fb4"}
Oct 30 07:32:48.291443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.291325    6738 scope.go:122] "RemoveContainer" containerID="726d974006f7ca47d44c129e9d62979d0c098975be8d7bf3d9107b859bc804a3"
Oct 30 07:32:48.360466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.360423    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:32:48.364490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.364455    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kwlpn"]
Oct 30 07:32:48.588743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:48.588644    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:49.697027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:49.696992    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" path="/var/lib/kubelet/pods/b52155c4-081e-4709-b6a6-6b31eeb6c7d7/volumes"
Oct 30 07:32:50.381552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:50.381514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:52.159784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.159750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:52.557082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.557053    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:32:52.599488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.599463    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:32:52.630556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630520    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.630775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.630775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630768    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.630916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630811    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.630916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630845    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.630916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630888    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.631085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.630931    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.631085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.631020    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.631209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.631085    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\") pod \"612fdfbc-d52e-45b3-9d55-67448f490e4a\" (UID: \"612fdfbc-d52e-45b3-9d55-67448f490e4a\") "
Oct 30 07:32:52.632332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.631970    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:32:52.632857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.632830    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.633582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.633427    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.633686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.631346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.634327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.633799    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.635336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.635222    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.635336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.635259    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.635336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.635272    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.635555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.635286    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.635678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.635563    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612fdfbc-d52e-45b3-9d55-67448f490e4a-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.639184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.639016    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "kube-api-access-6gcdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:52.645443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.645409    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:52.648602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.648548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.650350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.648930    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo" pod "612fdfbc-d52e-45b3-9d55-67448f490e4a" (UID: "612fdfbc-d52e-45b3-9d55-67448f490e4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:32:52.736503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736415    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736607    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736639    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736678    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736771    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736807    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736872    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.736926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736903    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.737142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.736982    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\") pod \"4b14404c-2018-4874-a3da-e61218f4166a\" (UID: \"4b14404c-2018-4874-a3da-e61218f4166a\") "
Oct 30 07:32:52.737142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.737142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737072    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.737541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737481    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.737920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737896    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737926    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737942    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737957    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737970    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcdz\" (UniqueName: \"kubernetes.io/projected/612fdfbc-d52e-45b3-9d55-67448f490e4a-kube-api-access-6gcdz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737986    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.737999    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612fdfbc-d52e-45b3-9d55-67448f490e4a-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.738522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.738496    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:32:52.738839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.738795    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.739145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.739122    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:32:52.739145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.739131    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:52.739397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.739380    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "kube-api-access-zrl6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:52.739632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.739619    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy" pod "4b14404c-2018-4874-a3da-e61218f4166a" (UID: "4b14404c-2018-4874-a3da-e61218f4166a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:52.839728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839694    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.839728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839725    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.839728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839734    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14404c-2018-4874-a3da-e61218f4166a-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.839994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839745    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrl6w\" (UniqueName: \"kubernetes.io/projected/4b14404c-2018-4874-a3da-e61218f4166a-kube-api-access-zrl6w\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.839994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839757    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14404c-2018-4874-a3da-e61218f4166a-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:52.839994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:52.839766    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14404c-2018-4874-a3da-e61218f4166a-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:32:53.332002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.331966    6738 generic.go:356] "Generic (PLEG): container finished" podID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerID="200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0" exitCode=0
Oct 30 07:32:53.332508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.332034    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"
Oct 30 07:32:53.332508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.332041    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" event={"ID":"612fdfbc-d52e-45b3-9d55-67448f490e4a","Type":"ContainerDied","Data":"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"}
Oct 30 07:32:53.332508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.332100    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z" event={"ID":"612fdfbc-d52e-45b3-9d55-67448f490e4a","Type":"ContainerDied","Data":"2e7274a372932d9708a3e8d9e38a8d0b7b5d137061fa4845a48032296c80e61b"}
Oct 30 07:32:53.332508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.332124    6738 scope.go:122] "RemoveContainer" containerID="200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"
Oct 30 07:32:53.334089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.334064    6738 generic.go:356] "Generic (PLEG): container finished" podID="4b14404c-2018-4874-a3da-e61218f4166a" containerID="eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d" exitCode=0
Oct 30 07:32:53.334257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.334198    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" event={"ID":"4b14404c-2018-4874-a3da-e61218f4166a","Type":"ContainerDied","Data":"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"}
Oct 30 07:32:53.334257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.334247    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn" event={"ID":"4b14404c-2018-4874-a3da-e61218f4166a","Type":"ContainerDied","Data":"0c56718c6704f5da4ed08e499da1654f473a53f4a82d5157b2c709c5666a4137"}
Oct 30 07:32:53.334515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.334274    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"
Oct 30 07:32:53.344556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.344473    6738 scope.go:122] "RemoveContainer" containerID="200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"
Oct 30 07:32:53.344862 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:53.344836    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0\": container with ID starting with 200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0 not found: ID does not exist" containerID="200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"
Oct 30 07:32:53.344931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.344873    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0"} err="failed to get container status \"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0\": rpc error: code = NotFound desc = could not find container \"200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0\": container with ID starting with 200c20880bed32e3adf7c51fb01a056871f181507a3406b1502b9bf2f08494c0 not found: ID does not exist"
Oct 30 07:32:53.344931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.344896    6738 scope.go:122] "RemoveContainer" containerID="eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"
Oct 30 07:32:53.355595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.355563    6738 scope.go:122] "RemoveContainer" containerID="eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"
Oct 30 07:32:53.355890 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:53.355865    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d\": container with ID starting with eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d not found: ID does not exist" containerID="eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"
Oct 30 07:32:53.355965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.355899    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d"} err="failed to get container status \"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d\": rpc error: code = NotFound desc = could not find container \"eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d\": container with ID starting with eac1c708daa66e0d0d073caa48f819c264fab53e5db76bbe7877e709ba527e7d not found: ID does not exist"
Oct 30 07:32:53.404517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.404475    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:32:53.417098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.417066    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8sdpn"]
Oct 30 07:32:53.449575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.449533    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:32:53.459587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.459555    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-8w76z"]
Oct 30 07:32:53.692111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.692077    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b14404c-2018-4874-a3da-e61218f4166a" path="/var/lib/kubelet/pods/4b14404c-2018-4874-a3da-e61218f4166a/volumes"
Oct 30 07:32:53.692698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:53.692682    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" path="/var/lib/kubelet/pods/612fdfbc-d52e-45b3-9d55-67448f490e4a/volumes"
Oct 30 07:32:54.076352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:54.076235    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:55.945919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.926012    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970393    6738 cpu_manager.go:414] "containerMap: removing container" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970432    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970473    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970482    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970524    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970532    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970788    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="612fdfbc-d52e-45b3-9d55-67448f490e4a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970807    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b14404c-2018-4874-a3da-e61218f4166a" containerName="istio-proxy"
Oct 30 07:32:55.973260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.970824    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52155c4-081e-4709-b6a6-6b31eeb6c7d7" containerName="istio-proxy"
Oct 30 07:32:55.986812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:55.979949    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.008482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.004546    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8e7948-serving-cert\""
Oct 30 07:32:56.096318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.095995    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 07:32:56.129676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.129631    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:56.130806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.130312    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.130806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.130371    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9zh\" (UniqueName: \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.130806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.130473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.130806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.130588    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.273372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.244424    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.273372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.244670    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.273372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.244711    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9zh\" (UniqueName: \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.273372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.244815    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.276338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.276285    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:56.277423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.277397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.278470 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:56.278448    6738 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-serving-cert: secret "maas-api-e2e-extmod-a-8e7948-serving-cert" not found
Oct 30 07:32:56.278697 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:32:56.278684    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls podName:45566de8-c39d-4c27-9cf0-3afd7822c95e nodeName:}" failed. No retries permitted until 2026-10-30 07:32:56.778649325 +0000 UTC m=+2137.211413203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls") pod "maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" (UID: "45566de8-c39d-4c27-9cf0-3afd7822c95e") : secret "maas-api-e2e-extmod-a-8e7948-serving-cert" not found
Oct 30 07:32:56.361174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.360114    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9zh\" (UniqueName: \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.393131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.379913    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 07:32:56.465330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.462739    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.486318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.479814    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 07:32:56.564456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.557644    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:32:56.835417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.834706    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.839097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.839062    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:56.998317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:56.997774    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:57.493337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:57.484161    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 07:32:58.437694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.437646    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" event={"ID":"45566de8-c39d-4c27-9cf0-3afd7822c95e","Type":"ContainerStarted","Data":"8c95d21e3eb5b1c88ca303ee3ca57c01efdf43ee067c23376f65dbe78d9eec44"}
Oct 30 07:32:58.438202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.437722    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:32:58.438202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.437741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" event={"ID":"45566de8-c39d-4c27-9cf0-3afd7822c95e","Type":"ContainerStarted","Data":"a81118683a4fffaa0311564e4be5c67b190162d32c1947f78f07ac95c0ed43fb"}
Oct 30 07:32:58.530338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.518052    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" podStartSLOduration=2.956777764 podStartE2EDuration="3.518023184s" totalImagesPullingTime="561.24542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:58.498718146 +0000 UTC m=+2138.931482046" watchObservedRunningTime="2026-10-30 07:32:58.518023184 +0000 UTC m=+2138.950787102"
Oct 30 07:32:58.552405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.550319    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:32:58.688867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:32:58.687899    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:00.822316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:00.821487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:02.764585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:02.764549    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:04.452977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:04.452943    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 07:33:04.667196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:04.667162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:06.517057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.517022    6738 generic.go:356] "Generic (PLEG): container finished" podID="eac308aa-bb24-44b4-8350-a94926d276b9" containerID="d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de" exitCode=0
Oct 30 07:33:06.517687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.517106    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerDied","Data":"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"}
Oct 30 07:33:06.517764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.517718    6738 scope.go:122] "RemoveContainer" containerID="d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"
Oct 30 07:33:06.519198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519175    6738 generic.go:356] "Generic (PLEG): container finished" podID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerID="f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e" exitCode=0
Oct 30 07:33:06.519348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519238    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerDied","Data":"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"}
Oct 30 07:33:06.519666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519648    6738 scope.go:122] "RemoveContainer" containerID="f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"
Oct 30 07:33:06.519777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519750    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.519841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519819    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.519892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.519855    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.521335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.521305    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.521460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.521385    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.521460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.521437    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:06.603755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:06.602789    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:07.532246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:07.532211    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerStarted","Data":"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"}
Oct 30 07:33:07.534765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:07.534731    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerStarted","Data":"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"}
Oct 30 07:33:07.546222 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:33:07.546187    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:33:08.061630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.061594    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:33:08.063357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.063323    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body=
Oct 30 07:33:08.063500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.063390    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused"
Oct 30 07:33:08.144771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.144671    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:33:08.144992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.144890    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 07:33:08.144992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.144946    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 07:33:08.529882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.529846    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:33:08.807995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:08.807901    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:09.062340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:09.062239    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body=
Oct 30 07:33:09.062506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:09.062333    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused"
Oct 30 07:33:09.145057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:09.145012    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 07:33:09.145246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:09.145073    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 07:33:10.062394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:10.062354    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body=
Oct 30 07:33:10.063049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:10.063011    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused"
Oct 30 07:33:10.145315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:10.145269    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 07:33:10.145532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:10.145422    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 07:33:11.062366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.062328    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body=
Oct 30 07:33:11.062552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.062396    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused"
Oct 30 07:33:11.108039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.108000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:11.155704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.151732    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 07:33:11.155704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.151796    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 07:33:11.942468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:11.942430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:12.061960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:12.061923    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body=
Oct 30 07:33:12.062144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:12.061978    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused"
Oct 30 07:33:12.145178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:12.145147    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 07:33:12.145754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:12.145727    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 07:33:13.067040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.067009    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:33:13.068258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.068230    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 07:33:13.149961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.149932    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:33:13.150404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.150228    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:33:13.150929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.150912    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:33:13.351674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.351571    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:13.471993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.471950    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:13.839347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.839260    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerName="main" containerID="cri-o://3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f" gracePeriod=45
Oct 30 07:33:13.911050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:13.911006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:14.190276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.190227    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerName="main" probeResult="failure" output="Get \"https://10.133.1.29:8000/ready\": dial tcp 10.133.1.29:8000: connect: connection refused"
Oct 30 07:33:14.253163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.253120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:14.330411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.330381    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:33:14.424769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.424722    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.424994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.424783    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\" (UniqueName: \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.424994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.424906    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.425115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425041    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:14.425115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425073    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\" (UniqueName: \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.425240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425130    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.425240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425154    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:14.425240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425209    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\") pod \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\" (UID: \"47eaf0fa-b417-4d9c-a08e-cbc2a188c418\") "
Oct 30 07:33:14.425479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425456    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:14.425933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425907    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.426048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425939    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.426048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.425957    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.427339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.427281    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:14.427438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.427359    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "kube-api-access-4f2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:14.427438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.427428    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs" pod "47eaf0fa-b417-4d9c-a08e-cbc2a188c418" (UID: "47eaf0fa-b417-4d9c-a08e-cbc2a188c418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:33:14.527406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.527321    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.527406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.527350    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.527406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.527361    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2lh\" (UniqueName: \"kubernetes.io/projected/47eaf0fa-b417-4d9c-a08e-cbc2a188c418-kube-api-access-4f2lh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:14.643811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.643776    6738 generic.go:356] "Generic (PLEG): container finished" podID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerID="3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f" exitCode=0
Oct 30 07:33:14.644001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.643821    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"
Oct 30 07:33:14.644001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.643854    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" event={"ID":"47eaf0fa-b417-4d9c-a08e-cbc2a188c418","Type":"ContainerDied","Data":"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"}
Oct 30 07:33:14.644001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.643897    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5" event={"ID":"47eaf0fa-b417-4d9c-a08e-cbc2a188c418","Type":"ContainerDied","Data":"f7f2382c695f871f91227abc3ad37ba8f0002d64aef40e7348502ee5cb2be6a5"}
Oct 30 07:33:14.644001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.643916    6738 scope.go:122] "RemoveContainer" containerID="3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"
Oct 30 07:33:14.657100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.657081    6738 scope.go:122] "RemoveContainer" containerID="3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"
Oct 30 07:33:14.657470 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:33:14.657448    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f\": container with ID starting with 3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f not found: ID does not exist" containerID="3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"
Oct 30 07:33:14.657562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.657481    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f"} err="failed to get container status \"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f\": rpc error: code = NotFound desc = could not find container \"3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f\": container with ID starting with 3a3e1c1ea068014d0640eefcaffc7082f7eae504878a6748de66d3be242c143f not found: ID does not exist"
Oct 30 07:33:14.698570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.698531    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:33:14.710764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:14.710732    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bd8589/auth-test-model-bd8589-kserve-854bf69c65-g4xk5"]
Oct 30 07:33:15.454815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:15.454783    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:15.692216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:15.692172    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" path="/var/lib/kubelet/pods/47eaf0fa-b417-4d9c-a08e-cbc2a188c418/volumes"
Oct 30 07:33:17.564315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:17.561408    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:18.009743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:18.009707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:18.162579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:18.162542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:18.301132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:18.301041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:19.051169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.051128    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:19.470438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.470403    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:19.682241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.682203    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:19.797729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.797641    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:19.875835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.875801    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 07:33:19.876632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.876611    6738 cpu_manager.go:414] "containerMap: removing container" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerName="main"
Oct 30 07:33:19.876632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.876631    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerName="main"
Oct 30 07:33:19.876799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.876765    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="47eaf0fa-b417-4d9c-a08e-cbc2a188c418" containerName="main"
Oct 30 07:33:19.880379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.880357    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:19.895718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.895686    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1f34aa-openshift-default-dockercfg-wdclw\""
Oct 30 07:33:19.912229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.912197    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 07:33:19.966028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:19.965989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:20.023829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.023788    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.023842    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.023958    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024030    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024165    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024342    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmb5\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024416    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.024560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.024522    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.094474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.094379    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:20.151919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.151876    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmb5\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.152266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.152239    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.152458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.152439    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.152614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.152595    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.152814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.152786    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.153396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.153374    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.154709    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.155351    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.155918    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.185436    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.221899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmb5\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.314320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.256683    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:20.328201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.314692    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.328201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.314802    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.328201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.314943    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.328201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.315677    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.328201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.317180    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.359451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.359364    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:20.369023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.359544    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.369023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.361158    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.369023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.367626    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\") pod \"e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") " pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.380314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.380224    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-25f44e-openshift-default-dockercfg-8jw4r\""
Oct 30 07:33:20.381526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.381417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416129    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416199    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416260    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416313    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416363    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416416    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416444    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwqk\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416617    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.418517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.416660    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.499770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.499730    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:20.517769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.517738    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.517930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.517782    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.517930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.517816    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.517931    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.517993    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518057    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518082    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwqk\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518188    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518247    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518324    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518337    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518396    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.518995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.518977    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.520826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.520806    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.520943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.520896    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.542349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.542310    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.542516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.542317    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwqk\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\") pod \"e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") " pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.659072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.659031    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:20.757506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.757419    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:20.834498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.832464    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 07:33:20.845767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.845713    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:20.845922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.845827    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:20.845922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:20.845875    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:21.092509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.092463    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:21.094214 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:33:21.094188    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23721b7_8475_451d_9a2c_f5f65f1a218e.slice/crio-df461f772c05850a9f53ff108ec93b1cefa91a97fa0b291d44b16e751adde590 WatchSource:0}: Error finding container df461f772c05850a9f53ff108ec93b1cefa91a97fa0b291d44b16e751adde590: Status 404 returned error can't find the container with id df461f772c05850a9f53ff108ec93b1cefa91a97fa0b291d44b16e751adde590
Oct 30 07:33:21.097414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.097384    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:21.097848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.097719    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:21.097848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.097807    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:21.098035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.097851    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:21.202178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.202145    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 07:33:21.206783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.206743    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.239847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.224891    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ace8fb\""
Oct 30 07:33:21.239847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.226727    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 07:33:21.239847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.228434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:21.256426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.248376    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.256426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.248596    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjd9\" (UniqueName: \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.419380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.418420    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.438321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.423746    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjd9\" (UniqueName: \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.534317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.520934    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.557669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.555514    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjd9\" (UniqueName: \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\") pod \"payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.561346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.560555    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:21.724724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.724288    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" event={"ID":"e23721b7-8475-451d-9a2c-f5f65f1a218e","Type":"ContainerStarted","Data":"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"}
Oct 30 07:33:21.724724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.724443    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" event={"ID":"e23721b7-8475-451d-9a2c-f5f65f1a218e","Type":"ContainerStarted","Data":"df461f772c05850a9f53ff108ec93b1cefa91a97fa0b291d44b16e751adde590"}
Oct 30 07:33:21.726798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.726768    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerStarted","Data":"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"}
Oct 30 07:33:21.727003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.726991    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerStarted","Data":"2eb7f0e4a4c0ec7644b2ca98ef160031c9fe9fd2b467c828229e0de6e9d4a92b"}
Oct 30 07:33:21.752114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.751456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:21.758132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.758101    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:21.773243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.773154    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" podStartSLOduration=1.7731361620000001 podStartE2EDuration="1.773136162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:21.772265246 +0000 UTC m=+2162.205029141" watchObservedRunningTime="2026-10-30 07:33:21.773136162 +0000 UTC m=+2162.205900055"
Oct 30 07:33:21.820393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.820330    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podStartSLOduration=2.820314894 podStartE2EDuration="2.820314894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:21.817554673 +0000 UTC m=+2162.250318578" watchObservedRunningTime="2026-10-30 07:33:21.820314894 +0000 UTC m=+2162.253078784"
Oct 30 07:33:21.861880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.861841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:21.893780 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:33:21.893741    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cdc8c8_6be6_46ef_9f1a_ff7e2c6505a1.slice/crio-64eaae2847c60f6a88f93916962feeb724400cc83afb5cbfe84bbc5742589814 WatchSource:0}: Error finding container 64eaae2847c60f6a88f93916962feeb724400cc83afb5cbfe84bbc5742589814: Status 404 returned error can't find the container with id 64eaae2847c60f6a88f93916962feeb724400cc83afb5cbfe84bbc5742589814
Oct 30 07:33:21.894566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:21.894545    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 07:33:22.235976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.235853    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:22.354362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.354250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:22.500496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.500458    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:22.506679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.506650    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:22.700408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.700369    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:22.736478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.736418    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" event={"ID":"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1","Type":"ContainerStarted","Data":"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"}
Oct 30 07:33:22.736717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.736485    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" event={"ID":"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1","Type":"ContainerStarted","Data":"64eaae2847c60f6a88f93916962feeb724400cc83afb5cbfe84bbc5742589814"}
Oct 30 07:33:22.737195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.737162    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:22.737362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.737223    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:22.738187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.738167    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:33:22.763898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.763865    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:22.789013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:22.788959    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" podStartSLOduration=1.78894457 podStartE2EDuration="1.78894457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:22.785405147 +0000 UTC m=+2163.218169040" watchObservedRunningTime="2026-10-30 07:33:22.78894457 +0000 UTC m=+2163.221708482"
Oct 30 07:33:23.745307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:23.745264    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:23.746359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:23.746324    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:23.938716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:23.938682    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:24.454627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:24.454595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:24.580228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:24.580180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:26.059655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.059604    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:26.238096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.238048    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:26.358319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.358220    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 07:33:26.362109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.362079    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.364247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.364211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:26.389232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.388366    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:33:26.398959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.398917    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 07:33:26.449085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.449041    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jwn\" (UniqueName: \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.449320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.449268    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.574492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.574443    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jwn\" (UniqueName: \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.574974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.574949    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.576190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.576163    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.616449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.616367    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jwn\" (UniqueName: \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\") pod \"payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:26.701360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:26.698348    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:27.021110 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:33:27.021064    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcba59f_2e73_455b_b7a2_416e1f7ad1bd.slice/crio-12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243 WatchSource:0}: Error finding container 12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243: Status 404 returned error can't find the container with id 12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243
Oct 30 07:33:27.021926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:27.021900    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 07:33:27.812380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:27.811194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" event={"ID":"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd","Type":"ContainerStarted","Data":"40284472999dff81a91eb3d1f79269d0c16cc228f17c1abfb1978e0cb6a1d6ba"}
Oct 30 07:33:27.812380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:27.811261    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" event={"ID":"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd","Type":"ContainerStarted","Data":"12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243"}
Oct 30 07:33:27.828698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:27.828373    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:27.907378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:27.907320    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" podStartSLOduration=1.907275099 podStartE2EDuration="1.907275099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:27.904050509 +0000 UTC m=+2168.336814403" watchObservedRunningTime="2026-10-30 07:33:27.907275099 +0000 UTC m=+2168.340038991"
Oct 30 07:33:28.230496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:28.230455    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:28.923657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:28.923615    6738 scope.go:122] "RemoveContainer" containerID="24d790b5a610bcead1ab2a62d54de99290cce20bf284b45c40cae705016cb0ec"
Oct 30 07:33:30.447826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:30.447783    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:31.968924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:31.968854    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:32.491318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:32.487102    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:34.467579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:34.467541    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:34.757309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:34.757216    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 07:33:36.371437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:36.371400    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:37.520079 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:33:37.520042    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:33:38.239881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.239842    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:38.953153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.953120    6738 generic.go:356] "Generic (PLEG): container finished" podID="95bf58b8-3623-494a-bb85-1b013279b919" containerID="e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384" exitCode=0
Oct 30 07:33:38.953655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.953192    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerDied","Data":"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"}
Oct 30 07:33:38.953655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.953648    6738 scope.go:122] "RemoveContainer" containerID="e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"
Oct 30 07:33:38.955705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.955674    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:38.955797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.955749    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:38.955797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:38.955783    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:33:39.870512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:39.870480    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 07:33:40.035512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.035461    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerStarted","Data":"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"}
Oct 30 07:33:40.195332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.195247    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:40.760084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.760047    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:40.792825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.792796    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:33:40.793108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.792862    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 07:33:40.793275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:40.793244    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 07:33:41.790543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:41.790504    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 07:33:41.791196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:41.791161    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 07:33:42.344025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:42.343990    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:42.460077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:42.460040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:42.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:42.790950    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 07:33:42.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:42.791026    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 07:33:43.787555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:43.787519    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 07:33:43.787868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:43.787840    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 07:33:43.822110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:43.822068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:44.409017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:44.408988    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:44.574262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:44.574216    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:44.720051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:44.719942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:44.797471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:44.797429    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 07:33:44.797989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:44.797956    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 07:33:45.349878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.349841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:45.541366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.541328    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:45.546799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.546768    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:45.547080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.547047    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" podUID="e23721b7-8475-451d-9a2c-f5f65f1a218e" containerName="istio-proxy" containerID="cri-o://27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3" gracePeriod=30
Oct 30 07:33:45.891792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.891754    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:45.965321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.965259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:33:45.965726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.965705    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:33:45.967426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:45.967394    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 07:33:46.072561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:46.072527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:46.593540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:46.593498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:48.554237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:48.554195    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:48.694205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:48.694154    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:50.304258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:50.304223    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:50.427155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:50.427115    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:50.959152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:50.959126    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:50.992836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:50.992803    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.052775    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.052848    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.052989    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053101    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053182    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053236    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053335    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.060318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.053433    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\") pod \"e23721b7-8475-451d-9a2c-f5f65f1a218e\" (UID: \"e23721b7-8475-451d-9a2c-f5f65f1a218e\") "
Oct 30 07:33:51.070144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.063107    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:51.070144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.065664    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:51.070144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.066219    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:51.070144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.066487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:51.072938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.070863    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:33:51.076120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.076032    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:33:51.076532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.076505    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:51.090101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.089968    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "kube-api-access-6wwqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:51.091802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.091765    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token" pod "e23721b7-8475-451d-9a2c-f5f65f1a218e" (UID: "e23721b7-8475-451d-9a2c-f5f65f1a218e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:51.103204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.103165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:51.155896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155847    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23721b7-8475-451d-9a2c-f5f65f1a218e-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.155896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155886    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.155896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155899    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwqk\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-kube-api-access-6wwqk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155912    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155924    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155934    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155947    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155958    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23721b7-8475-451d-9a2c-f5f65f1a218e-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.156184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.155969    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23721b7-8475-451d-9a2c-f5f65f1a218e-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:33:51.173755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.173718    6738 generic.go:356] "Generic (PLEG): container finished" podID="e23721b7-8475-451d-9a2c-f5f65f1a218e" containerID="27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3" exitCode=0
Oct 30 07:33:51.173949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.173778    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"
Oct 30 07:33:51.173949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.173804    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" event={"ID":"e23721b7-8475-451d-9a2c-f5f65f1a218e","Type":"ContainerDied","Data":"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"}
Oct 30 07:33:51.173949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.173861    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b" event={"ID":"e23721b7-8475-451d-9a2c-f5f65f1a218e","Type":"ContainerDied","Data":"df461f772c05850a9f53ff108ec93b1cefa91a97fa0b291d44b16e751adde590"}
Oct 30 07:33:51.173949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.173883    6738 scope.go:122] "RemoveContainer" containerID="27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"
Oct 30 07:33:51.186665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.186646    6738 scope.go:122] "RemoveContainer" containerID="27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"
Oct 30 07:33:51.186965 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:33:51.186948    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3\": container with ID starting with 27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3 not found: ID does not exist" containerID="27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"
Oct 30 07:33:51.187012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.186973    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3"} err="failed to get container status \"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3\": rpc error: code = NotFound desc = could not find container \"27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3\": container with ID starting with 27c54f05473a1d0a2b5df10c3a295537a572163f07ba8b21a7629b65669d44d3 not found: ID does not exist"
Oct 30 07:33:51.231626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.231582    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:51.237711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.237675    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-25f44e-openshift-default-76c474f4f-9vd5b"]
Oct 30 07:33:51.691527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:51.691490    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23721b7-8475-451d-9a2c-f5f65f1a218e" path="/var/lib/kubelet/pods/e23721b7-8475-451d-9a2c-f5f65f1a218e/volumes"
Oct 30 07:33:52.956041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:52.956009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:55.060898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:55.060858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:56.748038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:56.748002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:58.785057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:58.785023    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:58.911156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:58.911125    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:33:59.039357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:33:59.039258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:00.847095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:00.847057    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:02.712534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:02.712491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:04.673640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:04.673605    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:06.634211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:06.634176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:06.750255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:06.750204    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:06.865831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:06.865782    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:07.523487 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:34:07.523449    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:34:08.781741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:08.781695    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:08.920346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:08.915194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:10.107554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.107508    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:10.546861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.546828    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 07:34:10.547625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.547608    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e23721b7-8475-451d-9a2c-f5f65f1a218e" containerName="istio-proxy"
Oct 30 07:34:10.547694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.547626    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23721b7-8475-451d-9a2c-f5f65f1a218e" containerName="istio-proxy"
Oct 30 07:34:10.547795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.547786    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23721b7-8475-451d-9a2c-f5f65f1a218e" containerName="istio-proxy"
Oct 30 07:34:10.551780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.551754    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.585409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.585374    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 07:34:10.588879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.588848    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c1b5d7-openshift-default-dockercfg-dkplh\""
Oct 30 07:34:10.639782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.639739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:10.653268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653233    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.653475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653366    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.653475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653446    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.653847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653821    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.653946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653869    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.654011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.653983    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.654149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.654096    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.654241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.654222    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.654310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.654256    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj424\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755411    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755457    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755519    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755596    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755723    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj424\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.755832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755820    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755859    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755873    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755919    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755933    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.755973    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.756169    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.756462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.756442    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.758187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.758161    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.758422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.758403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.763059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.763033    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:10.780087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.780054    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj424\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.783770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.783746    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\") pod \"e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") " pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:10.877369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:10.874353    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:11.011008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.010949    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:11.363486 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:34:11.363453    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441fa007_4793_4b3c_b75b_51316af6d81e.slice/crio-1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b WatchSource:0}: Error finding container 1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b: Status 404 returned error can't find the container with id 1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b
Oct 30 07:34:11.364470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.364445    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 07:34:11.366456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.366421    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:11.366541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.366487    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:11.366541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.366519    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:11.418757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:11.418726    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerStarted","Data":"1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b"}
Oct 30 07:34:12.428674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:12.428634    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerStarted","Data":"057e748f8e403a9bf64537a809e0e95a0e590af58fd43af815c59b4f7f8c8a82"}
Oct 30 07:34:12.503582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:12.503520    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podStartSLOduration=2.503501317 podStartE2EDuration="2.503501317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:12.487011337 +0000 UTC m=+2212.919775230" watchObservedRunningTime="2026-10-30 07:34:12.503501317 +0000 UTC m=+2212.936265218"
Oct 30 07:34:12.756529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:12.756436    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:12.882198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:12.882165    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:12.888452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:12.888426    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:13.436982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:13.436945    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:13.438106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:13.438081    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:34:13.613628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:13.613593    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:14.676404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:14.676364    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:14.981456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:14.981367    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:15.140815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:15.140779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:16.699216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:16.699176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:16.828234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:16.828196    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:16.978601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:16.972126    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:18.735602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:18.735559    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:20.557530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:20.557492    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:22.460710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:22.460674    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:24.331506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:24.331466    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:26.246317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:26.244249    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:28.151193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:28.151152    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:28.398689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:28.398654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:29.991317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:29.990916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:30.158189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.158149    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:30.404367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.404323    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:34:30.416382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.416350    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.428697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.428667    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:34:30.438476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.433537    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8d0c2f-openshift-default-dockercfg-w2dlv\""
Oct 30 07:34:30.446491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446454    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt86\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446528    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446609    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446633    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446705    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446739    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446810    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.446851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446836    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.447071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.446859    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.453143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.453114    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:30.581386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.550022    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.583328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.582086    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.583592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.583569    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.584130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.584030    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.584401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.584381    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt86\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.584918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.584895    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.603319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.585566    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.604090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.604061    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.605189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.605167    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.605842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.605823    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.606441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.606419    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.626420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.606206    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.626420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.603984    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.626420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.605039    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.626420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.615576    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.644604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.641650    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.667478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.646734    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt86\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.667478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.646884    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\") pod \"e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") " pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:30.673317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.670453    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:30.739327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:30.735367    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:31.158403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.158371    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:34:31.158846 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:34:31.158774    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00511e0f_1eb2_4995_8455_b6618db0ed01.slice/crio-bb8f9d3754270d49b970b3c1e808877d0a08bc966bed411ec5ff7d1ff136aa06 WatchSource:0}: Error finding container bb8f9d3754270d49b970b3c1e808877d0a08bc966bed411ec5ff7d1ff136aa06: Status 404 returned error can't find the container with id bb8f9d3754270d49b970b3c1e808877d0a08bc966bed411ec5ff7d1ff136aa06
Oct 30 07:34:31.161786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.161750    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:31.161891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.161820    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:31.161891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.161851    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:31.706954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.706915    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerStarted","Data":"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"}
Oct 30 07:34:31.707154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.706984    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerStarted","Data":"bb8f9d3754270d49b970b3c1e808877d0a08bc966bed411ec5ff7d1ff136aa06"}
Oct 30 07:34:31.736623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.736592    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:31.784319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:31.768859    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podStartSLOduration=1.768841044 podStartE2EDuration="1.768841044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:31.765763202 +0000 UTC m=+2232.198527095" watchObservedRunningTime="2026-10-30 07:34:31.768841044 +0000 UTC m=+2232.201604953"
Oct 30 07:34:32.129148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:32.129113    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:32.249723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:32.249686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:32.400366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:32.399654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:32.744125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:32.744096    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:33.736479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:33.736441    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:33.741255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:33.741228    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:34:34.157801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:34.157763    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:34.277335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:34.277274    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:34.468446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:34.468335    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:35.952183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:35.952141    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:36.080242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.080207    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:36.143190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.143154    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 07:34:36.146964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.146943    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.167991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.167960    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 07:34:36.255521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.250806    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8d0c2f-dockercfg-rjszc\""
Oct 30 07:34:36.255521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.250908    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8d0c2f\""
Oct 30 07:34:36.366317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.360257    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq5v\" (UniqueName: \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.366317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.360393    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.462584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.462129    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.462584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.462377    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq5v\" (UniqueName: \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.463285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.463259    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.487313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.487267    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq5v\" (UniqueName: \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\") pod \"payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:36.739238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.739198    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:36.786387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:36.784680    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:37.139247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.139219    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 07:34:37.177544 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:34:37.177503    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efdbb2a_df9f_433f_9a02_057101bbf985.slice/crio-6214ddca4a9fa36732ce1228d85c3e0fd514c39b50a781d8a407685243c489af WatchSource:0}: Error finding container 6214ddca4a9fa36732ce1228d85c3e0fd514c39b50a781d8a407685243c489af: Status 404 returned error can't find the container with id 6214ddca4a9fa36732ce1228d85c3e0fd514c39b50a781d8a407685243c489af
Oct 30 07:34:37.520486 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:34:37.520449    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:34:37.740714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.740678    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:37.740947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.740729    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:37.773692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.773601    6738 generic.go:356] "Generic (PLEG): container finished" podID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerID="2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63" exitCode=0
Oct 30 07:34:37.773872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.773687    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerDied","Data":"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"}
Oct 30 07:34:37.774218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.774199    6738 scope.go:122] "RemoveContainer" containerID="2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"
Oct 30 07:34:37.775543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.775522    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" event={"ID":"7efdbb2a-df9f-433f-9a02-057101bbf985","Type":"ContainerStarted","Data":"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"}
Oct 30 07:34:37.775647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.775566    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" event={"ID":"7efdbb2a-df9f-433f-9a02-057101bbf985","Type":"ContainerStarted","Data":"6214ddca4a9fa36732ce1228d85c3e0fd514c39b50a781d8a407685243c489af"}
Oct 30 07:34:37.775706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.775665    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:37.776279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.776247    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:37.776391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.776357    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:37.776462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.776414    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:34:37.876977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:37.876495    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" podStartSLOduration=1.876475681 podStartE2EDuration="1.876475681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:37.875176751 +0000 UTC m=+2238.307940644" watchObservedRunningTime="2026-10-30 07:34:37.876475681 +0000 UTC m=+2238.309239575"
Oct 30 07:34:38.659820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:38.659780    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:38.788362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:38.788325    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerStarted","Data":"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"}
Oct 30 07:34:39.500380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:39.500343    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:34:39.500595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:39.500575    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:39.500644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:39.500618    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:40.500399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:40.500359    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:40.500833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:40.500422    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:40.658588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:40.658551    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:41.500047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:41.500013    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:41.500245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:41.500066    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:42.500379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:42.500337    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:42.500838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:42.500408    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:42.653151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:42.653112    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:43.500311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:43.500263    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:43.500518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:43.500348    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:44.501320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:44.501252    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 07:34:44.501842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:44.501330    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 07:34:44.792752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:44.792622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:45.545847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:45.545813    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:34:45.548148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:45.548121    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:34:45.552702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:45.552671    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 07:34:46.903076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:46.903033    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:48.791562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:48.791532    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 07:34:48.874334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:48.874255    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:50.419164 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:50.419128    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:50.739170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:50.739072    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:51.608484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:51.608446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:51.785718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:51.785686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:51.992688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:51.992646    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:52.237177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:52.237133    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:52.387879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:52.387836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:52.511414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:52.511373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:52.679155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:52.679054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:53.249940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:53.249910    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:53.367930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:53.367894    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:54.808213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:54.808175    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:56.677194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:56.677153    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:56.865390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:56.865270    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:57.919899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:57.919863    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:58.045634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:58.045592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:58.228885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:58.228788    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:58.573142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:58.573046    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:58.885850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:58.885817    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:59.038893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:59.038807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:59.574078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:59.574042    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:34:59.707803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:34:59.707757    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:01.099500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.099448    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:01.226635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.226584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:01.358902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.358822    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:35:01.361472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.361408    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:01.364137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.364117    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.382784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.382753    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:35:01.390049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.390012    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"openshift-service-ca.crt\""
Oct 30 07:35:01.390219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.390162    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-e2e--e56a4edc-dockercfg-s85b7\""
Oct 30 07:35:01.390265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.390225    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-dd6c04d9ae3b\""
Oct 30 07:35:01.390341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.390279    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:35:01.390435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.390420    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"routing-overlay\""
Oct 30 07:35:01.392712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.392681    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"kube-root-ca.crt\""
Oct 30 07:35:01.439792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.439748    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.440015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.439869    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.440015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.439927    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.440015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.439945    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsl2\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.552423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.552374    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.553282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.553252    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.553537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.553518    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.553678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.553661    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsl2\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.554113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.554088    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.568171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.568088    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.607376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.599616    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsl2\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.632307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.632254    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:01.690321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:01.680844    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:02.042314 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:35:02.039754    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca04fd2_20b6_4a06_b780_199d0c74c8e5.slice/crio-e75bf213802eddc71ab2bcb6d0d59e8ba82955a62f8348dbbf5cc827b0401575 WatchSource:0}: Error finding container e75bf213802eddc71ab2bcb6d0d59e8ba82955a62f8348dbbf5cc827b0401575: Status 404 returned error can't find the container with id e75bf213802eddc71ab2bcb6d0d59e8ba82955a62f8348dbbf5cc827b0401575
Oct 30 07:35:02.042314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:02.040351    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:35:02.083670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:02.083633    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" event={"ID":"7ca04fd2-20b6-4a06-b780-199d0c74c8e5","Type":"ContainerStarted","Data":"e75bf213802eddc71ab2bcb6d0d59e8ba82955a62f8348dbbf5cc827b0401575"}
Oct 30 07:35:03.094678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:03.094639    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" event={"ID":"7ca04fd2-20b6-4a06-b780-199d0c74c8e5","Type":"ContainerStarted","Data":"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"}
Oct 30 07:35:03.095096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:03.094698    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:03.137210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:03.137127    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" podStartSLOduration=2.13709145 podStartE2EDuration="2.13709145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:03.136514727 +0000 UTC m=+2263.569278615" watchObservedRunningTime="2026-10-30 07:35:03.13709145 +0000 UTC m=+2263.569855343"
Oct 30 07:35:03.302319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:03.296260    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:05.025110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:05.025073    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:06.950252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:06.950209    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:07.521186 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:35:07.521142    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:35:09.091496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:09.086989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:10.828815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:10.828778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:12.689696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:12.689656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:14.106958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:14.106926    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:35:14.595597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:14.595562    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:16.238428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:16.238384    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:16.365178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:16.365145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:16.536849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:16.536747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:18.549416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:18.549376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:20.643248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:20.643211    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:22.704109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:22.704068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:22.831318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:22.830234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:25.141035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:25.141000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:25.393536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:25.393446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:25.912105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:25.912070    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:27.142144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:27.142105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:27.313835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:27.313801    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:27.446405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:27.446282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:27.805914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:27.805821    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:27.950024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:27.949952    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:28.072979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:28.072886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:28.637734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:28.637688    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:28.753647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:28.753613    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:29.033097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:29.032453    6738 scope.go:122] "RemoveContainer" containerID="270aa5edf1d8b4890363d9a607d96c419096ee7110378d2d0c2a9ff050c59367"
Oct 30 07:35:29.445269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:29.445232    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:29.484467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:29.484425    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:35:29.484820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:29.484775    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:35:30.154468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:30.154426    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:31.417717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.417682    6738 generic.go:356] "Generic (PLEG): container finished" podID="441fa007-4793-4b3c-b75b-51316af6d81e" containerID="057e748f8e403a9bf64537a809e0e95a0e590af58fd43af815c59b4f7f8c8a82" exitCode=0
Oct 30 07:35:31.418162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.417745    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerDied","Data":"057e748f8e403a9bf64537a809e0e95a0e590af58fd43af815c59b4f7f8c8a82"}
Oct 30 07:35:31.418211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.418188    6738 scope.go:122] "RemoveContainer" containerID="057e748f8e403a9bf64537a809e0e95a0e590af58fd43af815c59b4f7f8c8a82"
Oct 30 07:35:31.420484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.420447    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:31.420617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.420537    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:31.420617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.420589    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:31.642396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:31.642342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:32.264707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:32.264665    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:32.429317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:32.429267    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerStarted","Data":"a87e8fad2404ab059324a0a1cb7dc3b159b27d7a34e20768696829a3e3b13d58"}
Oct 30 07:35:32.884824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:32.884784    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:35:32.885265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:32.885166    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:32.885524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:32.885499    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:33.561514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:33.561479    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:33.882852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:33.882816    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:33.883138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:33.883108    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:34.888855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:34.888808    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:34.889545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:34.889514    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:35.592323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:35.590621    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:35.887381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:35.886780    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:35.887381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:35.886842    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:36.884040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:36.882757    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:36.884040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:36.882838    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:37.434913 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:37.434875    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:37.600247 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:35:37.600209    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:35:37.885883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:37.885038    6738 patch_prober.go:29] interesting pod/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 30 07:35:37.885883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:37.885106    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 30 07:35:38.887908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:38.887877    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:35:38.888427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:38.888265    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:35:38.888935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:38.888913    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 07:35:39.584220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:39.584184    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:41.564219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:41.564182    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:43.499381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:43.499345    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:45.565690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:45.565658    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:47.529875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:47.529831    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:49.373355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:49.373319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:49.778552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:49.778456    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:35:49.778552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:49.778523    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:35:50.582269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.582234    6738 generic.go:356] "Generic (PLEG): container finished" podID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerID="cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552" exitCode=0
Oct 30 07:35:50.582835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.582309    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerDied","Data":"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"}
Oct 30 07:35:50.582905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.582865    6738 scope.go:122] "RemoveContainer" containerID="cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"
Oct 30 07:35:50.584937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.584905    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:50.585026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.584980    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:50.585026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:50.585012    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:35:51.333060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:51.333017    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:51.599774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:51.599678    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerStarted","Data":"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"}
Oct 30 07:35:51.736339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:51.736286    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:35:51.736559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:51.736498    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:51.736559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:51.736549    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:52.736655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:52.736613    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:52.737113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:52.736677    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:53.560951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:53.560903    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:53.735998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:53.735962    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:53.736195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:53.736028    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:54.736871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:54.736824    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:54.737330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:54.736896    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:55.695331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:55.695166    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:55.758367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:55.758328    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:55.759656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:55.759506    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:56.736147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:56.736114    6738 patch_prober.go:29] interesting pod/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 30 07:35:56.736474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:56.736449    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 30 07:35:57.219650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:57.219615    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:57.635274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:57.635242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:57.741966 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:57.741936    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:35:57.742284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:57.742259    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:35:57.743042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:57.743023    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:35:58.373934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:58.373897    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:58.491101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:58.491066    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:58.748351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:58.748259    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:58.977784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:58.977741    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:59.139491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:59.139458    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:59.286258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:59.286220    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:59.636018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:59.635989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:59.785059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:59.784973    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:35:59.902512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:35:59.902426    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:01.248705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:01.248665    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:01.800766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:01.800656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:03.727425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:03.727385    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:05.627326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:05.627275    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:07.519603 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:36:07.519560    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:36:07.542066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:07.542030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:09.321434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:09.321390    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:11.138813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:11.138772    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:12.956838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:12.956797    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:13.086191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:13.086146    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"]
Oct 30 07:36:13.091983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:13.091945    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fae8551-6hrx2"]
Oct 30 07:36:13.693482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:13.693447    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b83d00-e8c7-4a35-b4f9-3c21a9c71257" path="/var/lib/kubelet/pods/61b83d00-e8c7-4a35-b4f9-3c21a9c71257/volumes"
Oct 30 07:36:15.119143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:15.119099    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:16.455448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:16.455397    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:17.265969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:17.265909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:17.897817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:17.897767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:18.037511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:18.037470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:18.417594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:18.417552    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:18.536749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:18.536706    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:19.001482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.001446    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"]
Oct 30 07:36:19.005276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.005247    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.023392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.023363    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"]
Oct 30 07:36:19.030905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.030825    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:36:19.030905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.030887    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:36:19.095701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.095660    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499xd\" (UniqueName: \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.095897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.095814    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.095897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.095865    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.108222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.108190    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:19.197643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.197605    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.197878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.197855    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499xd\" (UniqueName: \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.198024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.198005    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.198668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.198639    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.200066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.200049    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.222160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.222080    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-499xd\" (UniqueName: \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\") pod \"maas-api-revoke-sub-77387f9c-mpvr2\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.232770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.232730    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:19.320342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.320228    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:19.450086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.449999    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:19.630337 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:36:19.630289    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6937f39_19a5_4ec8_9b83_358534fd1a79.slice/crio-d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1 WatchSource:0}: Error finding container d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1: Status 404 returned error can't find the container with id d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1
Oct 30 07:36:19.630775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:19.630748    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"]
Oct 30 07:36:20.007615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:20.007569    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2" event={"ID":"a6937f39-19a5-4ec8-9b83-358534fd1a79","Type":"ContainerStarted","Data":"d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1"}
Oct 30 07:36:21.017430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:21.017337    6738 generic.go:356] "Generic (PLEG): container finished" podID="a6937f39-19a5-4ec8-9b83-358534fd1a79" containerID="fa7e3219133ab53c40133b87791ef88b524b7bd4bd850dd3db31efa99f56ca2c" exitCode=0
Oct 30 07:36:21.017798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:21.017436    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2" event={"ID":"a6937f39-19a5-4ec8-9b83-358534fd1a79","Type":"ContainerDied","Data":"fa7e3219133ab53c40133b87791ef88b524b7bd4bd850dd3db31efa99f56ca2c"}
Oct 30 07:36:21.313672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:21.313578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:22.191518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.191493    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:22.223821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.223785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:22.361556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.361459    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\") pod \"a6937f39-19a5-4ec8-9b83-358534fd1a79\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") "
Oct 30 07:36:22.361752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.361563    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\" (UniqueName: \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\") pod \"a6937f39-19a5-4ec8-9b83-358534fd1a79\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") "
Oct 30 07:36:22.361752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.361713    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\") pod \"a6937f39-19a5-4ec8-9b83-358534fd1a79\" (UID: \"a6937f39-19a5-4ec8-9b83-358534fd1a79\") "
Oct 30 07:36:22.362087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.362054    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca" pod "a6937f39-19a5-4ec8-9b83-358534fd1a79" (UID: "a6937f39-19a5-4ec8-9b83-358534fd1a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:36:22.362366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.362347    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:22.363643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.363619    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca" pod "a6937f39-19a5-4ec8-9b83-358534fd1a79" (UID: "a6937f39-19a5-4ec8-9b83-358534fd1a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:36:22.363768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.363710    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd" pod "a6937f39-19a5-4ec8-9b83-358534fd1a79" (UID: "a6937f39-19a5-4ec8-9b83-358534fd1a79"). InnerVolumeSpecName "kube-api-access-499xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:36:22.386145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.386113    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:22.464744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.464707    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6937f39-19a5-4ec8-9b83-358534fd1a79-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:22.464744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:22.464743    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-499xd\" (UniqueName: \"kubernetes.io/projected/a6937f39-19a5-4ec8-9b83-358534fd1a79-kube-api-access-499xd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:23.034872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.034844    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"
Oct 30 07:36:23.035063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.034881    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2" event={"ID":"a6937f39-19a5-4ec8-9b83-358534fd1a79","Type":"ContainerDied","Data":"d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1"}
Oct 30 07:36:23.035063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.034929    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52e87bf58b2220083e788ea3a986519f4ff672f21b121975ee7643d7bc1eda1"
Oct 30 07:36:23.126742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.126712    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"]
Oct 30 07:36:23.130489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.130461    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e41aa52-84lgz"]
Oct 30 07:36:23.457040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.457004    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:23.699883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:23.699846    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10bcbeb-3b94-41b7-a28f-ccb8cc4de118" path="/var/lib/kubelet/pods/a10bcbeb-3b94-41b7-a28f-ccb8cc4de118/volumes"
Oct 30 07:36:24.034446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.034410    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:24.151718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.151681    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:24.468281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.466350    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 07:36:24.481472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.479751    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a6937f39-19a5-4ec8-9b83-358534fd1a79" containerName="revoke-keys"
Oct 30 07:36:24.481472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.479788    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6937f39-19a5-4ec8-9b83-358534fd1a79" containerName="revoke-keys"
Oct 30 07:36:24.481472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.480033    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6937f39-19a5-4ec8-9b83-358534fd1a79" containerName="revoke-keys"
Oct 30 07:36:24.499875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.499840    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 07:36:24.500374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.500001    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.521680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.521636    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e27e0c\""
Oct 30 07:36:24.531209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.531126    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e27e0c-dockercfg-k9b7t\""
Oct 30 07:36:24.652534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.652484    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.652727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.652646    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9fg\" (UniqueName: \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.754591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.754503    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.754758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.754606    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9fg\" (UniqueName: \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.755357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.755336    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.779057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.779017    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9fg\" (UniqueName: \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\") pod \"payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:24.842808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:24.842772    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:25.151761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:25.151730    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 07:36:25.151882 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:36:25.151850    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e932aa_c64e_4390_87f7_b59305f4641b.slice/crio-60ddf915cb3ab4a7d89b2b6aca2d0cd7f623fbb3db6b351929585c7881ddec5d WatchSource:0}: Error finding container 60ddf915cb3ab4a7d89b2b6aca2d0cd7f623fbb3db6b351929585c7881ddec5d: Status 404 returned error can't find the container with id 60ddf915cb3ab4a7d89b2b6aca2d0cd7f623fbb3db6b351929585c7881ddec5d
Oct 30 07:36:25.545987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:25.545900    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:26.061471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:26.061435    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" event={"ID":"e4e932aa-c64e-4390-87f7-b59305f4641b","Type":"ContainerStarted","Data":"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"}
Oct 30 07:36:26.061659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:26.061498    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" event={"ID":"e4e932aa-c64e-4390-87f7-b59305f4641b","Type":"ContainerStarted","Data":"60ddf915cb3ab4a7d89b2b6aca2d0cd7f623fbb3db6b351929585c7881ddec5d"}
Oct 30 07:36:26.061659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:26.061529    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:26.104920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:26.104862    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" podStartSLOduration=2.104847141 podStartE2EDuration="2.104847141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:26.102882287 +0000 UTC m=+2346.535646182" watchObservedRunningTime="2026-10-30 07:36:26.104847141 +0000 UTC m=+2346.537611033"
Oct 30 07:36:27.390709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:27.390666    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:29.246211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:29.246115    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:30.401039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:30.401002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:30.513917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:30.513885    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:31.055879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:31.055841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:31.343781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:31.343694    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:31.476922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:31.476890    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:36:32.382615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:32.382576    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:33.128166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:33.128125    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:33.854717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:33.854683    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:35.163171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:35.163132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:36.477527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.477483    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" podUID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" containerName="payload-processing" containerID="cri-o://8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5" gracePeriod=25
Oct 30 07:36:36.770174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.770146    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:36:36.856366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.856334    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\") pod \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") "
Oct 30 07:36:36.856585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.856453    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\") pod \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") "
Oct 30 07:36:36.856585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.856491    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\") pod \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") "
Oct 30 07:36:36.856747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.856582    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\") pod \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\" (UID: \"7ca04fd2-20b6-4a06-b780-199d0c74c8e5\") "
Oct 30 07:36:36.857066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.857030    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume" pod "7ca04fd2-20b6-4a06-b780-199d0c74c8e5" (UID: "7ca04fd2-20b6-4a06-b780-199d0c74c8e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:36:36.857483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.857439    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:36.857483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.857455    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume" pod "7ca04fd2-20b6-4a06-b780-199d0c74c8e5" (UID: "7ca04fd2-20b6-4a06-b780-199d0c74c8e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:36:36.859186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.859153    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials" pod "7ca04fd2-20b6-4a06-b780-199d0c74c8e5" (UID: "7ca04fd2-20b6-4a06-b780-199d0c74c8e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:36:36.859332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.859284    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2" pod "7ca04fd2-20b6-4a06-b780-199d0c74c8e5" (UID: "7ca04fd2-20b6-4a06-b780-199d0c74c8e5"). InnerVolumeSpecName "kube-api-access-hmsl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:36:36.958655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.958621    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:36.958655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.958651    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:36.958655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:36.958662    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsl2\" (UniqueName: \"kubernetes.io/projected/7ca04fd2-20b6-4a06-b780-199d0c74c8e5-kube-api-access-hmsl2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:36:37.049395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.049274    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:37.073584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.073558    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 07:36:37.309371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.304143    6738 generic.go:356] "Generic (PLEG): container finished" podID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" containerID="8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5" exitCode=0
Oct 30 07:36:37.309371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.304496    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" event={"ID":"7ca04fd2-20b6-4a06-b780-199d0c74c8e5","Type":"ContainerDied","Data":"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"}
Oct 30 07:36:37.309371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.304575    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg" event={"ID":"7ca04fd2-20b6-4a06-b780-199d0c74c8e5","Type":"ContainerDied","Data":"e75bf213802eddc71ab2bcb6d0d59e8ba82955a62f8348dbbf5cc827b0401575"}
Oct 30 07:36:37.309371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.304618    6738 scope.go:122] "RemoveContainer" containerID="8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"
Oct 30 07:36:37.309371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.304839    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"
Oct 30 07:36:37.343608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.343582    6738 scope.go:122] "RemoveContainer" containerID="8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"
Oct 30 07:36:37.344017 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:36:37.343986    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5\": container with ID starting with 8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5 not found: ID does not exist" containerID="8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"
Oct 30 07:36:37.344138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.344027    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5"} err="failed to get container status \"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5\": rpc error: code = NotFound desc = could not find container \"8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5\": container with ID starting with 8d05e4ca3d384210e8584fb32f1d4bbebd04c820e84c42d39a4a602036d25bc5 not found: ID does not exist"
Oct 30 07:36:37.375033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.375000    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:36:37.378893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.378863    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc44hpkg"]
Oct 30 07:36:37.414993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.414956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:37.535933 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:36:37.535894    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:36:37.546171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.546141    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:37.659353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.659321    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:37.698438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:37.698398    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" path="/var/lib/kubelet/pods/7ca04fd2-20b6-4a06-b780-199d0c74c8e5/volumes"
Oct 30 07:36:38.941579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:38.941537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:40.853147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:40.853107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:42.662628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:42.662581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:44.668319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:44.663547    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:44.947082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:44.946988    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:45.229436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:45.229344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:45.390574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:45.390542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:46.405743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:46.405704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:46.862842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:46.862751    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:47.043384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:47.043333    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"]
Oct 30 07:36:47.078315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:47.075887    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019a3e9-5mqls"]
Oct 30 07:36:47.691809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:47.691775    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d" path="/var/lib/kubelet/pods/8a1d4e4d-79ef-42b0-a3d0-a7d8af86899d/volumes"
Oct 30 07:36:47.967004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:47.966831    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:48.116403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:48.116364    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:49.046425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.046364    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"]
Oct 30 07:36:49.051512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.051474    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ffa48c-ptpb6"]
Oct 30 07:36:49.483131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.483096    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:36:49.484028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.484004    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" containerName="payload-processing"
Oct 30 07:36:49.484028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.484027    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" containerName="payload-processing"
Oct 30 07:36:49.484247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.484233    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca04fd2-20b6-4a06-b780-199d0c74c8e5" containerName="payload-processing"
Oct 30 07:36:49.487375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.487351    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.508439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.508403    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:36:49.516066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516036    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"openshift-service-ca.crt\""
Oct 30 07:36:49.516264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516106    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-dd6c04d9ae3b\""
Oct 30 07:36:49.516264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516177    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"routing-overlay\""
Oct 30 07:36:49.516264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516208    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:36:49.516264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516217    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-652ba7d1390f\""
Oct 30 07:36:49.516559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516373    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-e2e--e56a4edc-dockercfg-s85b7\""
Oct 30 07:36:49.516559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516440    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-61d3b8b09e9a\""
Oct 30 07:36:49.516559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.516118    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"kube-root-ca.crt\""
Oct 30 07:36:49.589037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.589000    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.589037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.589042    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcm9\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.589306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.589113    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.589306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.589225    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.631687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.631656    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:49.696695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.696625    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.712348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.712286    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcm9\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.712677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.712654    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.712959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.712933    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.713917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.713507    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.720998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.719600    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.735771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.735677    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.737348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.737286    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcm9\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:49.742974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.742945    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e34c02b-b2bf-41ef-b34b-5f24194a9ab1" path="/var/lib/kubelet/pods/2e34c02b-b2bf-41ef-b34b-5f24194a9ab1/volumes"
Oct 30 07:36:49.802441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:49.802405    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:50.116793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.116762    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:36:50.117166 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:36:50.116982    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84bc3fe_0477_4013_a4e6_5bd5190e9ff5.slice/crio-8791b3d90a3588dd3db6632690024d19ca4fe99312bbad79ec96e352692eb718 WatchSource:0}: Error finding container 8791b3d90a3588dd3db6632690024d19ca4fe99312bbad79ec96e352692eb718: Status 404 returned error can't find the container with id 8791b3d90a3588dd3db6632690024d19ca4fe99312bbad79ec96e352692eb718
Oct 30 07:36:50.467343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.467289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:50.470129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.470099    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" event={"ID":"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5","Type":"ContainerStarted","Data":"243ea894062da895260a22a403a7d9b66cb0ff154bec110b009253ca6709dc49"}
Oct 30 07:36:50.470309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.470149    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" event={"ID":"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5","Type":"ContainerStarted","Data":"8791b3d90a3588dd3db6632690024d19ca4fe99312bbad79ec96e352692eb718"}
Oct 30 07:36:50.470309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.470181    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:36:50.515702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:50.515649    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" podStartSLOduration=1.515635637 podStartE2EDuration="1.515635637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:50.512100754 +0000 UTC m=+2370.944864641" watchObservedRunningTime="2026-10-30 07:36:50.515635637 +0000 UTC m=+2370.948399529"
Oct 30 07:36:51.230325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.229621    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:36:51.261117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.260130    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.284459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.284428    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:36:51.287675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.287649    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"agc-cred-0506141fea2e\""
Oct 30 07:36:51.300793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.300756    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-e2e--97d1e837-dockercfg-4q8ml\""
Oct 30 07:36:51.315819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.315654    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"routing-overlay\""
Oct 30 07:36:51.315819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.315661    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8f1df\""
Oct 30 07:36:51.315819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.315751    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"openshift-service-ca.crt\""
Oct 30 07:36:51.315819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.315803    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"kube-root-ca.crt\""
Oct 30 07:36:51.364408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.364368    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.364408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.364408    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.364679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.364589    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.364679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.364623    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmwh\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.465868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.465833    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.465868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.465869    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.466132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.465958    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.466132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.465983    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmwh\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.466698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.466662    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.466829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.466772    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.468425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.468403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.490203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.490129    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmwh\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.599745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.599707    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:51.692906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.692850    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:51.911908 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:36:51.911876    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7933b6a6_8683_40c5_88cc_39cf713064f8.slice/crio-0b6d8683aee204c77f91ab5c9553262e636c98b95367682ed4dab4ca6986f77f WatchSource:0}: Error finding container 0b6d8683aee204c77f91ab5c9553262e636c98b95367682ed4dab4ca6986f77f: Status 404 returned error can't find the container with id 0b6d8683aee204c77f91ab5c9553262e636c98b95367682ed4dab4ca6986f77f
Oct 30 07:36:51.912411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:51.912389    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:36:52.489377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:52.489340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" event={"ID":"7933b6a6-8683-40c5-88cc-39cf713064f8","Type":"ContainerStarted","Data":"40bf383dcc19f5a22ea57ce41516dcf3168b1916c4e8ba22f03912c86c94f601"}
Oct 30 07:36:52.489797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:52.489403    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" event={"ID":"7933b6a6-8683-40c5-88cc-39cf713064f8","Type":"ContainerStarted","Data":"0b6d8683aee204c77f91ab5c9553262e636c98b95367682ed4dab4ca6986f77f"}
Oct 30 07:36:52.489797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:52.489433    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:52.533593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:52.533540    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" podStartSLOduration=1.53352511 podStartE2EDuration="1.53352511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:52.531374355 +0000 UTC m=+2372.964138249" watchObservedRunningTime="2026-10-30 07:36:52.53352511 +0000 UTC m=+2372.966289013"
Oct 30 07:36:53.662092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:53.662060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:55.612248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:55.612210    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:56.351786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:56.351749    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:36:56.354956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:56.354930    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:36:56.434896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:56.434862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:57.033514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:57.033483    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"]
Oct 30 07:36:57.045005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:57.044971    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07442568-dw4s2"]
Oct 30 07:36:57.547881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:57.547850    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:36:57.692909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:57.692876    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39cb3feb-ee20-4ecb-8645-fbc4f6096aaf" path="/var/lib/kubelet/pods/39cb3feb-ee20-4ecb-8645-fbc4f6096aaf/volumes"
Oct 30 07:36:59.361862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:36:59.361832    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:00.031747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:00.031708    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"]
Oct 30 07:37:00.043387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:00.043352    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02708227-mt22m"]
Oct 30 07:37:01.321112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.321059    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:01.352471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.352431    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" podUID="7933b6a6-8683-40c5-88cc-39cf713064f8" containerName="payload-processing" containerID="cri-o://40bf383dcc19f5a22ea57ce41516dcf3168b1916c4e8ba22f03912c86c94f601" gracePeriod=25
Oct 30 07:37:01.484244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.484213    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:37:01.567347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.567257    6738 generic.go:356] "Generic (PLEG): container finished" podID="7933b6a6-8683-40c5-88cc-39cf713064f8" containerID="40bf383dcc19f5a22ea57ce41516dcf3168b1916c4e8ba22f03912c86c94f601" exitCode=0
Oct 30 07:37:01.567347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.567338    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" event={"ID":"7933b6a6-8683-40c5-88cc-39cf713064f8","Type":"ContainerDied","Data":"40bf383dcc19f5a22ea57ce41516dcf3168b1916c4e8ba22f03912c86c94f601"}
Oct 30 07:37:01.656221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.656193    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:37:01.692844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.692805    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0" path="/var/lib/kubelet/pods/0be02b41-8bf8-4fb4-b0a2-f0e9fca269c0/volumes"
Oct 30 07:37:01.764775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.764739    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\") pod \"7933b6a6-8683-40c5-88cc-39cf713064f8\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") "
Oct 30 07:37:01.764990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.764969    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\") pod \"7933b6a6-8683-40c5-88cc-39cf713064f8\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") "
Oct 30 07:37:01.765065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765028    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\") pod \"7933b6a6-8683-40c5-88cc-39cf713064f8\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") "
Oct 30 07:37:01.765139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765125    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\") pod \"7933b6a6-8683-40c5-88cc-39cf713064f8\" (UID: \"7933b6a6-8683-40c5-88cc-39cf713064f8\") "
Oct 30 07:37:01.765178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765138    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume" pod "7933b6a6-8683-40c5-88cc-39cf713064f8" (UID: "7933b6a6-8683-40c5-88cc-39cf713064f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:01.765554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765530    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume" pod "7933b6a6-8683-40c5-88cc-39cf713064f8" (UID: "7933b6a6-8683-40c5-88cc-39cf713064f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:01.765762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765741    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:01.765831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.765767    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7933b6a6-8683-40c5-88cc-39cf713064f8-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:01.767378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.767358    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials" pod "7933b6a6-8683-40c5-88cc-39cf713064f8" (UID: "7933b6a6-8683-40c5-88cc-39cf713064f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:01.767378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.767380    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh" pod "7933b6a6-8683-40c5-88cc-39cf713064f8" (UID: "7933b6a6-8683-40c5-88cc-39cf713064f8"). InnerVolumeSpecName "kube-api-access-czmwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:01.867779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.867745    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czmwh\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-kube-api-access-czmwh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:01.867779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:01.867777    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7933b6a6-8683-40c5-88cc-39cf713064f8-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:02.577504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:02.577465    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm" event={"ID":"7933b6a6-8683-40c5-88cc-39cf713064f8","Type":"ContainerDied","Data":"0b6d8683aee204c77f91ab5c9553262e636c98b95367682ed4dab4ca6986f77f"}
Oct 30 07:37:02.577504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:02.577496    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"
Oct 30 07:37:02.578093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:02.577527    6738 scope.go:122] "RemoveContainer" containerID="40bf383dcc19f5a22ea57ce41516dcf3168b1916c4e8ba22f03912c86c94f601"
Oct 30 07:37:02.633266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:02.633228    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:37:02.645672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:02.645643    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-6bbfw2ptm"]
Oct 30 07:37:03.140637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:03.140601    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:03.692416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:03.692378    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7933b6a6-8683-40c5-88cc-39cf713064f8" path="/var/lib/kubelet/pods/7933b6a6-8683-40c5-88cc-39cf713064f8/volumes"
Oct 30 07:37:04.981449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:04.981417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:06.911560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:06.911515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:07.687873 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:37:07.687832    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:37:08.962048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:08.962013    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:10.783982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:10.783941    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:10.898413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:10.898375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:11.013967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:11.013931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:11.329033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:11.328981    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:11.447391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:11.447359    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:12.331470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.331438    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:12.458868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.458836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:12.848285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.848255    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:37:12.849097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.849080    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7933b6a6-8683-40c5-88cc-39cf713064f8" containerName="payload-processing"
Oct 30 07:37:12.849158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.849098    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7933b6a6-8683-40c5-88cc-39cf713064f8" containerName="payload-processing"
Oct 30 07:37:12.849247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.849238    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7933b6a6-8683-40c5-88cc-39cf713064f8" containerName="payload-processing"
Oct 30 07:37:12.852863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.852842    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.862520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.862492    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.862643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.862621    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.862690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.862680    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.862734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.862707    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ht9\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.870680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.870652    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880469    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1f700\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880496    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-00a7322d0aa7\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880507    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-2c0ab0fb80d7\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880563    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"kube-root-ca.crt\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880574    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"openshift-service-ca.crt\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880604    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"routing-overlay\""
Oct 30 07:37:12.885344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.880669    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-e2e--96fc5bc0-dockercfg-tlgdn\""
Oct 30 07:37:12.964224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.964187    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.964453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.964266    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.964453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.964337    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.964453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.964371    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ht9\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.965182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.965149    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.965263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.965248    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.967169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.967142    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:12.988607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:12.988577    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ht9\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:13.020498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.020462    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:13.173458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.173420    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:13.503112 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:37:13.503074    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ff34c8_f82a_4e47_874b_f969f0d63dff.slice/crio-2bae3bb7f1d679343175690607b587eb2cca28c26f3b5f6de3f061a2a5919143 WatchSource:0}: Error finding container 2bae3bb7f1d679343175690607b587eb2cca28c26f3b5f6de3f061a2a5919143: Status 404 returned error can't find the container with id 2bae3bb7f1d679343175690607b587eb2cca28c26f3b5f6de3f061a2a5919143
Oct 30 07:37:13.505325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.505283    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:37:13.796193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.796077    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" event={"ID":"85ff34c8-f82a-4e47-874b-f969f0d63dff","Type":"ContainerStarted","Data":"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"}
Oct 30 07:37:13.796193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.796138    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" event={"ID":"85ff34c8-f82a-4e47-874b-f969f0d63dff","Type":"ContainerStarted","Data":"2bae3bb7f1d679343175690607b587eb2cca28c26f3b5f6de3f061a2a5919143"}
Oct 30 07:37:13.796193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.796166    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:13.842160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:13.842103    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" podStartSLOduration=1.8420866249999999 podStartE2EDuration="1.842086625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:13.838138998 +0000 UTC m=+2394.270902892" watchObservedRunningTime="2026-10-30 07:37:13.842086625 +0000 UTC m=+2394.274850559"
Oct 30 07:37:14.848466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:14.848419    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:16.685507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:16.685474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:17.431666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:17.431627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:17.562811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:17.562775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:18.080810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.080771    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"]
Oct 30 07:37:18.084616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.084595    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.117199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.117162    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"]
Oct 30 07:37:18.128365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.128336    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:37:18.134588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.134553    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:37:18.154997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.154947    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nlx\" (UniqueName: \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.155253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.155229    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.155373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.155328    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.256553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.256518    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.256757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.256580    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.256757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.256721    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nlx\" (UniqueName: \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.257235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.257210    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.259163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.259139    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.293226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.293195    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nlx\" (UniqueName: \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\") pod \"maas-api-revoke-sub-c51b1640-g8pmf\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.399583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.399545    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:18.716308 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:37:18.716254    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d71610_9154_4b6a_8cd5_4e3e644c095b.slice/crio-c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08 WatchSource:0}: Error finding container c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08: Status 404 returned error can't find the container with id c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08
Oct 30 07:37:18.717059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.716963    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"]
Oct 30 07:37:18.839241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.839203    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf" event={"ID":"64d71610-9154-4b6a-8cd5-4e3e644c095b","Type":"ContainerStarted","Data":"c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08"}
Oct 30 07:37:18.849374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:18.849338    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:19.850963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:19.850928    6738 generic.go:356] "Generic (PLEG): container finished" podID="64d71610-9154-4b6a-8cd5-4e3e644c095b" containerID="969f4afe3e73badc58fd89ff45774a35cd325386cea407c8e69ac17f14b84c06" exitCode=0
Oct 30 07:37:19.851433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:19.851015    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf" event={"ID":"64d71610-9154-4b6a-8cd5-4e3e644c095b","Type":"ContainerDied","Data":"969f4afe3e73badc58fd89ff45774a35cd325386cea407c8e69ac17f14b84c06"}
Oct 30 07:37:20.824359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:20.824322    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:21.077107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.077029    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:21.100772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.100739    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\") pod \"64d71610-9154-4b6a-8cd5-4e3e644c095b\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") "
Oct 30 07:37:21.100956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.100882    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\") pod \"64d71610-9154-4b6a-8cd5-4e3e644c095b\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") "
Oct 30 07:37:21.101222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.101194    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\" (UniqueName: \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\") pod \"64d71610-9154-4b6a-8cd5-4e3e644c095b\" (UID: \"64d71610-9154-4b6a-8cd5-4e3e644c095b\") "
Oct 30 07:37:21.101395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.101249    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca" pod "64d71610-9154-4b6a-8cd5-4e3e644c095b" (UID: "64d71610-9154-4b6a-8cd5-4e3e644c095b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:21.101803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.101779    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:21.103609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.103582    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca" pod "64d71610-9154-4b6a-8cd5-4e3e644c095b" (UID: "64d71610-9154-4b6a-8cd5-4e3e644c095b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:37:21.103703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.103616    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx" pod "64d71610-9154-4b6a-8cd5-4e3e644c095b" (UID: "64d71610-9154-4b6a-8cd5-4e3e644c095b"). InnerVolumeSpecName "kube-api-access-54nlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:21.202997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.202952    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54nlx\" (UniqueName: \"kubernetes.io/projected/64d71610-9154-4b6a-8cd5-4e3e644c095b-kube-api-access-54nlx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:21.202997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.202987    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d71610-9154-4b6a-8cd5-4e3e644c095b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:21.870287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.870252    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"
Oct 30 07:37:21.870495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.870286    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf" event={"ID":"64d71610-9154-4b6a-8cd5-4e3e644c095b","Type":"ContainerDied","Data":"c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08"}
Oct 30 07:37:21.870495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:21.870356    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26955d74f55040ef64d8486e14aff9511d68f3c3cb6298872d3701b072e2b08"
Oct 30 07:37:22.032142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:22.032049    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"]
Oct 30 07:37:22.040232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:22.040199    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fde6cc8-wklrq"]
Oct 30 07:37:22.681565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:22.681527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:22.960852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:22.960760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:23.694900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:23.694869    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c" path="/var/lib/kubelet/pods/947cebf4-ad18-4c5c-8f25-6bfbc4e9c76c/volumes"
Oct 30 07:37:24.530480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:24.530437    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:24.807997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:24.807919    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:37:26.435638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:26.435596    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:28.294096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:28.294056    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:29.116207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:29.116174    6738 scope.go:122] "RemoveContainer" containerID="8b299d060d33573221b23ad0bedc7f4b7f24c8f27746a4bbc9c861e968e591e4"
Oct 30 07:37:29.127627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:29.127605    6738 scope.go:122] "RemoveContainer" containerID="1c99d01c222bf65fff58890edc33817058b12e89f26f626281670ecf8eda18e1"
Oct 30 07:37:29.567733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:29.567697    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:29.700644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:29.700592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:30.170675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:30.170639    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:30.552453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:30.552367    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:32.449276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:32.449239    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:32.561249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:32.561207    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:33.031449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:33.031415    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"]
Oct 30 07:37:33.043600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:33.043556    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1e9fa6-5p265"]
Oct 30 07:37:33.701597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:33.701557    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d95a29-96fb-4483-aba7-d7cab27d52ba" path="/var/lib/kubelet/pods/93d95a29-96fb-4483-aba7-d7cab27d52ba/volumes"
Oct 30 07:37:34.213418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:34.213378    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:36.054612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:36.054572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:36.454972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:36.454935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:36.581940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:36.581906    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:36.700698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:36.700661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:37.519724 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:37:37.519689    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:37:37.921641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:37.921606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:39.858320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:39.852107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:41.294787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:41.294750    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:37:41.295423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:41.294955    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy" containerID="cri-o://a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18" gracePeriod=30
Oct 30 07:37:41.886153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:41.886117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.005516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.005473    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.149806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.149774    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.315940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.315855    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.525576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.525542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.699239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.699204    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:43.939635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:43.938934    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:44.441956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:44.441923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:46.236471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.236438    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:46.597826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.597790    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:46.618960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.618933    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.771811    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772059    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772108    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772141    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772211    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772240    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772352    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772454    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.772499    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\") pod \"05e4f9bf-b9a2-4783-9021-053f34d7215d\" (UID: \"05e4f9bf-b9a2-4783-9021-053f34d7215d\") "
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.774505    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:46.777360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.774758    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:46.778137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.777671    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.785418    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:46.791319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.786680    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:46.839322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.838471    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "kube-api-access-dzv2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:46.839322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.839307    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:46.868388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.867212    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:46.868388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.867456    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo" pod "05e4f9bf-b9a2-4783-9021-053f34d7215d" (UID: "05e4f9bf-b9a2-4783-9021-053f34d7215d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879069    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879104    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879120    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879133    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e4f9bf-b9a2-4783-9021-053f34d7215d-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879145    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879158    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzv2r\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-kube-api-access-dzv2r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879171    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879186    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e4f9bf-b9a2-4783-9021-053f34d7215d-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:46.881663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:46.879198    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e4f9bf-b9a2-4783-9021-053f34d7215d-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:47.103340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.103215    6738 generic.go:356] "Generic (PLEG): container finished" podID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerID="a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18" exitCode=0
Oct 30 07:37:47.103340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.103277    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"
Oct 30 07:37:47.103340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.103304    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerDied","Data":"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"}
Oct 30 07:37:47.103711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.103360    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l" event={"ID":"05e4f9bf-b9a2-4783-9021-053f34d7215d","Type":"ContainerDied","Data":"c8fdba433d3ca151323ac9df65681bda45413071210426b9be28cca54ede6d7c"}
Oct 30 07:37:47.103711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.103395    6738 scope.go:122] "RemoveContainer" containerID="a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"
Oct 30 07:37:47.115510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.115493    6738 scope.go:122] "RemoveContainer" containerID="1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"
Oct 30 07:37:47.126933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.126911    6738 scope.go:122] "RemoveContainer" containerID="a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"
Oct 30 07:37:47.127285 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:37:47.127259    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18\": container with ID starting with a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18 not found: ID does not exist" containerID="a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"
Oct 30 07:37:47.127384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.127327    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18"} err="failed to get container status \"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18\": rpc error: code = NotFound desc = could not find container \"a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18\": container with ID starting with a6ee139c4c38f75fcfb4d0623dac25212da51af42c013df0e0dfb44a0368ce18 not found: ID does not exist"
Oct 30 07:37:47.127384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.127355    6738 scope.go:122] "RemoveContainer" containerID="1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"
Oct 30 07:37:47.127701 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:37:47.127681    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5\": container with ID starting with 1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5 not found: ID does not exist" containerID="1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"
Oct 30 07:37:47.127811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.127705    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5"} err="failed to get container status \"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5\": rpc error: code = NotFound desc = could not find container \"1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5\": container with ID starting with 1f1b22dd3748612a6f29b5d09a97a6fca00a3f6abedf2c7f10612a88bb106ca5 not found: ID does not exist"
Oct 30 07:37:47.159650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.159611    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:37:47.172394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.172363    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-zsm9l"]
Oct 30 07:37:47.445579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.445545    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:47.557415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.557374    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:47.694351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:47.694314    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" path="/var/lib/kubelet/pods/05e4f9bf-b9a2-4783-9021-053f34d7215d/volumes"
Oct 30 07:37:48.404746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:48.404689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:48.864798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:48.864699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:49.001319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:48.994447    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:37:49.001319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:48.994850    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy" containerID="cri-o://e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d" gracePeriod=30
Oct 30 07:37:50.568171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:50.568130    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:50.691424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:50.691352    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:50.844108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:50.844006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:51.515916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:51.515875    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:51.645904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:51.645871    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:53.065674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.065638    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:53.729440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.729407    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:37:53.730219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730203    6738 cpu_manager.go:414] "containerMap: removing container" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.730284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730221    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.730284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730244    6738 cpu_manager.go:414] "containerMap: removing container" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.730284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730249    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.730284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730277    6738 cpu_manager.go:414] "containerMap: removing container" podUID="64d71610-9154-4b6a-8cd5-4e3e644c095b" containerName="revoke-keys"
Oct 30 07:37:53.730284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730283    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d71610-9154-4b6a-8cd5-4e3e644c095b" containerName="revoke-keys"
Oct 30 07:37:53.730485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730417    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d71610-9154-4b6a-8cd5-4e3e644c095b" containerName="revoke-keys"
Oct 30 07:37:53.730485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730433    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.730485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.730442    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e4f9bf-b9a2-4783-9021-053f34d7215d" containerName="istio-proxy"
Oct 30 07:37:53.735620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.735589    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.753887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.753857    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:37:53.763130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.762347    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:37:53.763130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.762425    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-7acbf0effedd\""
Oct 30 07:37:53.763130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.762427    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:37:53.763130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.762454    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-2wsrt\""
Oct 30 07:37:53.763130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.762498    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-a7a7ea4f917f\""
Oct 30 07:37:53.763748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.763726    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:37:53.763894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.763766    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:37:53.845028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.844992    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.845266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.845232    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.845512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.845492    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltwk\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.845599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.845558    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.947136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.947090    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltwk\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.947366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.947158    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.947366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.947183    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.947366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.947289    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.948176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.948143    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.948338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.948239    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.949996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.949975    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:53.971371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:53.971336    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltwk\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:54.057380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.057255    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:54.366326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.366272    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:37:54.377842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.377809    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:37:54.378145 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:37:54.378123    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b86b66d_e43b_48d3_9b0c_ed6cd77a856b.slice/crio-e70f72b8017e4c933020549940f2de5ebf7d59a97058c2ca736f10c0e9908a76 WatchSource:0}: Error finding container e70f72b8017e4c933020549940f2de5ebf7d59a97058c2ca736f10c0e9908a76: Status 404 returned error can't find the container with id e70f72b8017e4c933020549940f2de5ebf7d59a97058c2ca736f10c0e9908a76
Oct 30 07:37:54.454695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.454664    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.454970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.454845    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.454970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.454911    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.454970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.454947    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.455500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.454981    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.455500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.455032    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.455500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.455074    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.455500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.455155    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.455500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.455195    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\") pod \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\" (UID: \"a01a0375-83a1-4f1d-b828-d424e6fa2dcf\") "
Oct 30 07:37:54.457614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.457538    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:54.458120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.457977    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:54.458120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.458092    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:54.458614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.458559    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:54.458975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.458946    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:54.460614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.460586    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:37:54.460791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.460758    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:37:54.460985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.460964    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "kube-api-access-k2jg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:54.461090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.461070    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token" pod "a01a0375-83a1-4f1d-b828-d424e6fa2dcf" (UID: "a01a0375-83a1-4f1d-b828-d424e6fa2dcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:54.557497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557459    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557491    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557502    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557512    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jg4\" (UniqueName: \"kubernetes.io/projected/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-kube-api-access-k2jg4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557521    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557530    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557539    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557547    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.557768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.557555    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01a0375-83a1-4f1d-b828-d424e6fa2dcf-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:37:54.996405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:54.996370    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:55.192527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.192494    6738 generic.go:356] "Generic (PLEG): container finished" podID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerID="e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d" exitCode=0
Oct 30 07:37:55.192743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.192544    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"
Oct 30 07:37:55.192743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.192573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" event={"ID":"a01a0375-83a1-4f1d-b828-d424e6fa2dcf","Type":"ContainerDied","Data":"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"}
Oct 30 07:37:55.192743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.192634    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b" event={"ID":"a01a0375-83a1-4f1d-b828-d424e6fa2dcf","Type":"ContainerDied","Data":"324d612576f061040a91be0815a96ea6b62210b6d4dfc1fca2449d73679c2780"}
Oct 30 07:37:55.192743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.192666    6738 scope.go:122] "RemoveContainer" containerID="e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"
Oct 30 07:37:55.195177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.195149    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" event={"ID":"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b","Type":"ContainerStarted","Data":"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"}
Oct 30 07:37:55.195315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.195203    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" event={"ID":"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b","Type":"ContainerStarted","Data":"e70f72b8017e4c933020549940f2de5ebf7d59a97058c2ca736f10c0e9908a76"}
Oct 30 07:37:55.195535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.195502    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:37:55.206978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.206958    6738 scope.go:122] "RemoveContainer" containerID="e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"
Oct 30 07:37:55.207331 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:37:55.207277    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d\": container with ID starting with e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d not found: ID does not exist" containerID="e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"
Oct 30 07:37:55.207414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.207340    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d"} err="failed to get container status \"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d\": rpc error: code = NotFound desc = could not find container \"e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d\": container with ID starting with e68e4e71f921e35625f156b25cbc20c8f9c1fbc6f526a2c9a26c13f143206e7d not found: ID does not exist"
Oct 30 07:37:55.254797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.254693    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" podStartSLOduration=2.2546764599999998 podStartE2EDuration="2.25467646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:55.252376877 +0000 UTC m=+2435.685140771" watchObservedRunningTime="2026-10-30 07:37:55.25467646 +0000 UTC m=+2435.687440353"
Oct 30 07:37:55.295545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.295488    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:37:55.311356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.311321    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-kbq7b"]
Oct 30 07:37:55.691897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:55.691864    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" path="/var/lib/kubelet/pods/a01a0375-83a1-4f1d-b828-d424e6fa2dcf/volumes"
Oct 30 07:37:56.859776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:56.859739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:58.743014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:58.742975    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:37:59.912174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:37:59.912140    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:00.599793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:00.599760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:02.566810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:02.566770    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:04.507469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:04.507428    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:06.208813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:06.208781    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:38:06.700230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:06.700194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:07.530968 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:07.530928    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:38:08.699397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:08.699361    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:09.287165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.287114    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"]
Oct 30 07:38:09.288038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.288014    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy"
Oct 30 07:38:09.288038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.288038    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy"
Oct 30 07:38:09.288207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.288177    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01a0375-83a1-4f1d-b828-d424e6fa2dcf" containerName="istio-proxy"
Oct 30 07:38:09.293412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.293393    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.321174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.321140    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:38:09.321424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.321185    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:38:09.326154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.326122    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"]
Oct 30 07:38:09.435952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.435917    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwlw\" (UniqueName: \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.436151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.436072    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.436151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.436107    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.537409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.537284    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwlw\" (UniqueName: \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.537565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.537447    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.537565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.537472    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.538191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.538167    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.539955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.539930    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.561460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.561424    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwlw\" (UniqueName: \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\") pod \"maas-api-revoke-sub-3504f37b-2kv4h\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.609697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.609658    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:09.800839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.800750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:09.919917 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:09.919866    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd04e5d_78a3_4b87_bfe3_fdbb7b963c51.slice/crio-24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf WatchSource:0}: Error finding container 24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf: Status 404 returned error can't find the container with id 24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf
Oct 30 07:38:09.920796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:09.920763    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"]
Oct 30 07:38:10.334770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:10.334733    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h" event={"ID":"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51","Type":"ContainerStarted","Data":"34d551980dd89c90ea6cd851e44a6f2af79735e977d12ae156eb3d047480901a"}
Oct 30 07:38:10.334968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:10.334793    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h" event={"ID":"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51","Type":"ContainerStarted","Data":"24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf"}
Oct 30 07:38:10.378196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:10.378118    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h" podStartSLOduration=1.378099581 podStartE2EDuration="1.378099581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:10.375501316 +0000 UTC m=+2450.808265234" watchObservedRunningTime="2026-10-30 07:38:10.378099581 +0000 UTC m=+2450.810863475"
Oct 30 07:38:10.644959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:10.644916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:11.120422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.120336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:11.239720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.239686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:11.344428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.344394    6738 generic.go:356] "Generic (PLEG): container finished" podID="9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" containerID="34d551980dd89c90ea6cd851e44a6f2af79735e977d12ae156eb3d047480901a" exitCode=0
Oct 30 07:38:11.344638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.344477    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h" event={"ID":"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51","Type":"ContainerDied","Data":"34d551980dd89c90ea6cd851e44a6f2af79735e977d12ae156eb3d047480901a"}
Oct 30 07:38:11.488231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.488197    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:11.795236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.795143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:11.963447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:11.963417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:12.083884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.083796    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:12.530015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.529987    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:12.597205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.597173    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\") pod \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") "
Oct 30 07:38:12.597434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.597414    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\") pod \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") "
Oct 30 07:38:12.597496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.597458    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\" (UniqueName: \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\") pod \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\" (UID: \"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51\") "
Oct 30 07:38:12.597960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.597928    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca" pod "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" (UID: "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:12.598243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.598223    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:12.600039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.600012    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca" pod "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" (UID: "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:12.600211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.600071    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw" pod "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" (UID: "9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51"). InnerVolumeSpecName "kube-api-access-ppwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:12.644068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.644036    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:12.700238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.700201    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppwlw\" (UniqueName: \"kubernetes.io/projected/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-kube-api-access-ppwlw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:12.700238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.700238    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:12.786650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.776836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:12.903574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:12.903540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:13.362988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:13.362898    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"
Oct 30 07:38:13.362988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:13.362924    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h" event={"ID":"9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51","Type":"ContainerDied","Data":"24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf"}
Oct 30 07:38:13.362988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:13.362978    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e9aabd52bf925be26cdb022c9e805450a89e9c4dd254645e23692c736e81bf"
Oct 30 07:38:14.831592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:14.831555    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:16.451632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.451596    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"]
Oct 30 07:38:16.452460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.452443    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" containerName="revoke-keys"
Oct 30 07:38:16.452523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.452460    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" containerName="revoke-keys"
Oct 30 07:38:16.452596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.452587    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" containerName="revoke-keys"
Oct 30 07:38:16.456371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.456350    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.474387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.474357    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"]
Oct 30 07:38:16.481994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.481962    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:38:16.482235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.482110    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:38:16.556950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.556896    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.557155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.557124    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694f5\" (UniqueName: \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.557262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.557246    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.658860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.658812    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.659051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.658923    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.659149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.659076    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694f5\" (UniqueName: \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.659740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.659713    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.661569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.661544    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.683659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.683623    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694f5\" (UniqueName: \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\") pod \"maas-api-revoke-sub-51ff70bc-tpsnz\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:16.690066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.690032    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:16.775212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:16.775129    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:17.090551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:17.090524    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"]
Oct 30 07:38:17.090691 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:17.090581    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115f63c2_4dcd_4e32_bbc8_0e2d7a00360f.slice/crio-8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5 WatchSource:0}: Error finding container 8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5: Status 404 returned error can't find the container with id 8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5
Oct 30 07:38:17.400110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:17.400067    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz" event={"ID":"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f","Type":"ContainerStarted","Data":"8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5"}
Oct 30 07:38:18.411351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:18.411308    6738 generic.go:356] "Generic (PLEG): container finished" podID="115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" containerID="fbacba39f877503f88c6d80105599b40c2ddc3c2b254fe123ba391d7550cde5a" exitCode=0
Oct 30 07:38:18.411775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:18.411393    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz" event={"ID":"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f","Type":"ContainerDied","Data":"fbacba39f877503f88c6d80105599b40c2ddc3c2b254fe123ba391d7550cde5a"}
Oct 30 07:38:18.547536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:18.547502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:19.584357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.584330    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:19.731277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.731193    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\") pod \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") "
Oct 30 07:38:19.731498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.731409    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\") pod \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") "
Oct 30 07:38:19.731498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.731454    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\" (UniqueName: \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\") pod \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\" (UID: \"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f\") "
Oct 30 07:38:19.731607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.731572    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca" pod "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" (UID: "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:19.732005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.731980    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:19.733727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.733706    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca" pod "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" (UID: "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:19.734157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.734130    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5" pod "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" (UID: "115f63c2-4dcd-4e32-bbc8-0e2d7a00360f"). InnerVolumeSpecName "kube-api-access-694f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:19.833346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.833276    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:19.833346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.833349    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694f5\" (UniqueName: \"kubernetes.io/projected/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f-kube-api-access-694f5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:19.870885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.870852    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"]
Oct 30 07:38:19.871853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.871829    6738 cpu_manager.go:414] "containerMap: removing container" podUID="115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" containerName="revoke-keys"
Oct 30 07:38:19.871853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.871850    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" containerName="revoke-keys"
Oct 30 07:38:19.872016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.872009    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" containerName="revoke-keys"
Oct 30 07:38:19.876084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.876060    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:19.893457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.893421    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"]
Oct 30 07:38:19.936115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.935907    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:19.936115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.936030    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:19.936115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.936069    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfq7\" (UniqueName: \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:19.943126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:19.943093    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:20.068986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.065448    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.068986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.065523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfq7\" (UniqueName: \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.068986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.065922    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.068986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.066812    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.070909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.070876    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.080129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.080091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:20.091327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.091281    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfq7\" (UniqueName: \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\") pod \"maas-api-revoke-sub-98e0cfca-cbstl\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.196770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.196715    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:20.430262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.430235    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"
Oct 30 07:38:20.430457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.430273    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz" event={"ID":"115f63c2-4dcd-4e32-bbc8-0e2d7a00360f","Type":"ContainerDied","Data":"8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5"}
Oct 30 07:38:20.430457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.430343    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8220fbe764fd54d2c5dc33985d2d24df1ef78d8e557bd22c9f635c0cc209b9e5"
Oct 30 07:38:20.505448 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:20.505413    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539d15bd_4568_4cf0_98e6_45af7a63376e.slice/crio-43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36 WatchSource:0}: Error finding container 43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36: Status 404 returned error can't find the container with id 43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36
Oct 30 07:38:20.506060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.506030    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"]
Oct 30 07:38:20.536775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.536744    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:20.702654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:20.702114    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:21.252373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.249139    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"]
Oct 30 07:38:21.346322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.335695    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"]
Oct 30 07:38:21.346322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.335828    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.399223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.399191    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.440135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.440098    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl" event={"ID":"539d15bd-4568-4cf0-98e6-45af7a63376e","Type":"ContainerStarted","Data":"2866d7051c1da4c420f0f6a77257388fefd38e64088cb55e7e2c218d1f09c381"}
Oct 30 07:38:21.440375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.440156    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl" event={"ID":"539d15bd-4568-4cf0-98e6-45af7a63376e","Type":"ContainerStarted","Data":"43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36"}
Oct 30 07:38:21.484587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.484532    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl" podStartSLOduration=2.484516841 podStartE2EDuration="2.484516841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:21.480997997 +0000 UTC m=+2461.913761890" watchObservedRunningTime="2026-10-30 07:38:21.484516841 +0000 UTC m=+2461.917280785"
Oct 30 07:38:21.501766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.501093    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.501766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.501166    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcx92\" (UniqueName: \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.501766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.501442    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.502077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.502057    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.602310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.602264    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.602519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.602332    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcx92\" (UniqueName: \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.604847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.604812    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.626922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.626887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcx92\" (UniqueName: \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\") pod \"maas-api-revoke-sub-f86289c6-dzlcg\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:21.699565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:21.699527    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:22.006286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.006259    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"]
Oct 30 07:38:22.006728 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:22.006393    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3ebb3f_6204_49e9_95d5_88a57e5aebc6.slice/crio-b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0 WatchSource:0}: Error finding container b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0: Status 404 returned error can't find the container with id b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0
Oct 30 07:38:22.468831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.468618    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg" event={"ID":"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6","Type":"ContainerStarted","Data":"22052d59e3e502e7ca7cc390ba37bf62bb4ffa2692b600e150f027387eef5ff0"}
Oct 30 07:38:22.469048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.468863    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg" event={"ID":"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6","Type":"ContainerStarted","Data":"b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0"}
Oct 30 07:38:22.471146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.471120    6738 generic.go:356] "Generic (PLEG): container finished" podID="539d15bd-4568-4cf0-98e6-45af7a63376e" containerID="2866d7051c1da4c420f0f6a77257388fefd38e64088cb55e7e2c218d1f09c381" exitCode=0
Oct 30 07:38:22.471320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.471148    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl" event={"ID":"539d15bd-4568-4cf0-98e6-45af7a63376e","Type":"ContainerDied","Data":"2866d7051c1da4c420f0f6a77257388fefd38e64088cb55e7e2c218d1f09c381"}
Oct 30 07:38:22.511824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.511772    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg" podStartSLOduration=1.511756479 podStartE2EDuration="1.511756479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:22.510017398 +0000 UTC m=+2462.942781292" watchObservedRunningTime="2026-10-30 07:38:22.511756479 +0000 UTC m=+2462.944520377"
Oct 30 07:38:22.646817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:22.646734    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:23.480635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.480599    6738 generic.go:356] "Generic (PLEG): container finished" podID="1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" containerID="22052d59e3e502e7ca7cc390ba37bf62bb4ffa2692b600e150f027387eef5ff0" exitCode=0
Oct 30 07:38:23.481119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.480680    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg" event={"ID":"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6","Type":"ContainerDied","Data":"22052d59e3e502e7ca7cc390ba37bf62bb4ffa2692b600e150f027387eef5ff0"}
Oct 30 07:38:23.660230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.660204    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:23.734876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.734781    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\") pod \"539d15bd-4568-4cf0-98e6-45af7a63376e\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") "
Oct 30 07:38:23.735059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.734922    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\" (UniqueName: \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\") pod \"539d15bd-4568-4cf0-98e6-45af7a63376e\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") "
Oct 30 07:38:23.735195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.735151    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca" pod "539d15bd-4568-4cf0-98e6-45af7a63376e" (UID: "539d15bd-4568-4cf0-98e6-45af7a63376e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:23.735273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.735251    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\") pod \"539d15bd-4568-4cf0-98e6-45af7a63376e\" (UID: \"539d15bd-4568-4cf0-98e6-45af7a63376e\") "
Oct 30 07:38:23.736000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.735974    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:23.737478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.737445    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7" pod "539d15bd-4568-4cf0-98e6-45af7a63376e" (UID: "539d15bd-4568-4cf0-98e6-45af7a63376e"). InnerVolumeSpecName "kube-api-access-7bfq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:23.738551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.738518    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca" pod "539d15bd-4568-4cf0-98e6-45af7a63376e" (UID: "539d15bd-4568-4cf0-98e6-45af7a63376e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:23.837029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.836998    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539d15bd-4568-4cf0-98e6-45af7a63376e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:23.837029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:23.837026    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfq7\" (UniqueName: \"kubernetes.io/projected/539d15bd-4568-4cf0-98e6-45af7a63376e-kube-api-access-7bfq7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:24.490197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.490161    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"
Oct 30 07:38:24.490660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.490194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl" event={"ID":"539d15bd-4568-4cf0-98e6-45af7a63376e","Type":"ContainerDied","Data":"43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36"}
Oct 30 07:38:24.490660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.490246    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43770086b58558994ce1f7f40bbf3d34c3755d22d7ace0231c4d1c061eed6a36"
Oct 30 07:38:24.516059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.516022    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:24.715927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.715898    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:24.748441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.748353    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\" (UniqueName: \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\") pod \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") "
Oct 30 07:38:24.748596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.748446    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\") pod \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") "
Oct 30 07:38:24.748596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.748488    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\") pod \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\" (UID: \"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6\") "
Oct 30 07:38:24.748873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.748850    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca" pod "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" (UID: "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:24.749125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.749109    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:24.750667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.750648    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca" pod "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" (UID: "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:24.750757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.750737    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92" pod "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" (UID: "1d3ebb3f-6204-49e9-95d5-88a57e5aebc6"). InnerVolumeSpecName "kube-api-access-bcx92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:24.850174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.850142    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcx92\" (UniqueName: \"kubernetes.io/projected/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-kube-api-access-bcx92\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:24.850174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:24.850172    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:25.500607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:25.500569    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg" event={"ID":"1d3ebb3f-6204-49e9-95d5-88a57e5aebc6","Type":"ContainerDied","Data":"b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0"}
Oct 30 07:38:25.500607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:25.500609    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"
Oct 30 07:38:25.501097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:25.500632    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b9c5f0fbad1b7e3720edd5f7fd72efa9e43bb5b8d051a4239ba77dd3a690a0"
Oct 30 07:38:26.351809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:26.351779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:27.154986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:27.154950    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:27.273403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:27.273354    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:27.385654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:27.385614    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:27.829846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:27.829811    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:27.952926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:27.952862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:28.153486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:28.153453    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:28.574063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:28.573958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:28.714833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:28.714798    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:28.869570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:28.869479    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:29.007872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.007836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:29.138360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.138322    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:29.177972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.177937    6738 scope.go:122] "RemoveContainer" containerID="6000344ea2d67552d270b46110798ddc6e17d750dcdd4aa2fda17cce10ad9df5"
Oct 30 07:38:29.188993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.188971    6738 scope.go:122] "RemoveContainer" containerID="25f7f7a3b864a0c9a8ab7157a3986489b901244b1d11806a81c7fbb47bd4ce3b"
Oct 30 07:38:29.199385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.199354    6738 scope.go:122] "RemoveContainer" containerID="bb06ff45edbfec357de935168ba6f1e3fabaa42ab490e751c34c7628339fee76"
Oct 30 07:38:29.209275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.209254    6738 scope.go:122] "RemoveContainer" containerID="1bc3987625afba7f2b512d4f9e0419e2ff7ae0f0948f221ab515ec419dda1de1"
Oct 30 07:38:29.218904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.218876    6738 scope.go:122] "RemoveContainer" containerID="7f7e142dcde093b57bc265e3dca13e251fffa61c0347ae56537378e85b559c92"
Oct 30 07:38:29.331941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.331904    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:29.963885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:29.963848    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:30.073692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:30.073657    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:31.028015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:31.027970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:31.355618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:31.355536    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:31.498399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:31.498366    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:32.271738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.271705    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"]
Oct 30 07:38:32.275429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.274749    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" containerName="revoke-keys"
Oct 30 07:38:32.276063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.276044    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" containerName="revoke-keys"
Oct 30 07:38:32.276326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.276279    6738 cpu_manager.go:414] "containerMap: removing container" podUID="539d15bd-4568-4cf0-98e6-45af7a63376e" containerName="revoke-keys"
Oct 30 07:38:32.276439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.276429    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539d15bd-4568-4cf0-98e6-45af7a63376e" containerName="revoke-keys"
Oct 30 07:38:32.276898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.276658    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="539d15bd-4568-4cf0-98e6-45af7a63376e" containerName="revoke-keys"
Oct 30 07:38:32.277074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.277059    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" containerName="revoke-keys"
Oct 30 07:38:32.288662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.283056    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"]
Oct 30 07:38:32.288662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.283177    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.298878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.298852    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:38:32.299021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.298968    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:38:32.365116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.365075    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.365116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.365119    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.365440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.365418    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8hz\" (UniqueName: \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.466501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.466463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8hz\" (UniqueName: \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.466688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.466542    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.466688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.466562    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.467261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.467242    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.468988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.468960    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.494227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.494196    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8hz\" (UniqueName: \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\") pod \"maas-api-revoke-sub-107fb25d-fg6kd\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.566116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.566001    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:32.600630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.600597    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:32.693468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.693432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:32.815675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.815639    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:32.928363 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:32.928332    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786eb420_ea6c_4025_b8e1_09d36e331662.slice/crio-86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3 WatchSource:0}: Error finding container 86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3: Status 404 returned error can't find the container with id 86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3
Oct 30 07:38:32.928989 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:32.928964    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"]
Oct 30 07:38:33.164194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.164160    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:33.515125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.515093    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:38:33.601662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.601556    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd" event={"ID":"786eb420-ea6c-4025-b8e1-09d36e331662","Type":"ContainerStarted","Data":"6836fd0d1ee0c567478598f451dae4cb7c46fb5c59f8809f0b4473780690f0d3"}
Oct 30 07:38:33.601662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.601621    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd" event={"ID":"786eb420-ea6c-4025-b8e1-09d36e331662","Type":"ContainerStarted","Data":"86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3"}
Oct 30 07:38:33.648216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.648160    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd" podStartSLOduration=1.648143735 podStartE2EDuration="1.648143735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:33.644084403 +0000 UTC m=+2474.076848311" watchObservedRunningTime="2026-10-30 07:38:33.648143735 +0000 UTC m=+2474.080907627"
Oct 30 07:38:33.700327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.699252    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:33.976319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:33.971223    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:34.348755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.348658    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"]
Oct 30 07:38:34.353522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.353493    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.373005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.372965    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"]
Oct 30 07:38:34.577051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.577011    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.577692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.577162    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxrr\" (UniqueName: \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.577692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.577210    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.620785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.620671    6738 generic.go:356] "Generic (PLEG): container finished" podID="786eb420-ea6c-4025-b8e1-09d36e331662" containerID="6836fd0d1ee0c567478598f451dae4cb7c46fb5c59f8809f0b4473780690f0d3" exitCode=0
Oct 30 07:38:34.620785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.620733    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd" event={"ID":"786eb420-ea6c-4025-b8e1-09d36e331662","Type":"ContainerDied","Data":"6836fd0d1ee0c567478598f451dae4cb7c46fb5c59f8809f0b4473780690f0d3"}
Oct 30 07:38:34.678970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.678930    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.679212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.679001    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxrr\" (UniqueName: \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.679212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.679048    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.679812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.679789    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.681592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.681555    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.704365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.704328    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxrr\" (UniqueName: \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\") pod \"maas-api-revoke-sub-fbf64ddc-w77qr\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:34.970006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:34.969970    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:35.277611 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:38:35.277582    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4482d977_9965_4541_9ccd_2535df24055b.slice/crio-77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71 WatchSource:0}: Error finding container 77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71: Status 404 returned error can't find the container with id 77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71
Oct 30 07:38:35.281094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.279512    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"]
Oct 30 07:38:35.469973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.469931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:35.592147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.591746    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:35.630356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.630318    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr" event={"ID":"4482d977-9965-4541-9ccd-2535df24055b","Type":"ContainerStarted","Data":"77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71"}
Oct 30 07:38:35.815263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.815237    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:35.873236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.873204    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:35.901273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.901236    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\") pod \"786eb420-ea6c-4025-b8e1-09d36e331662\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") "
Oct 30 07:38:35.901495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.901425    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\") pod \"786eb420-ea6c-4025-b8e1-09d36e331662\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") "
Oct 30 07:38:35.901640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.901621    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\" (UniqueName: \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\") pod \"786eb420-ea6c-4025-b8e1-09d36e331662\" (UID: \"786eb420-ea6c-4025-b8e1-09d36e331662\") "
Oct 30 07:38:35.901877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.901850    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca" pod "786eb420-ea6c-4025-b8e1-09d36e331662" (UID: "786eb420-ea6c-4025-b8e1-09d36e331662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:35.902222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.902201    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:35.903651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.903620    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca" pod "786eb420-ea6c-4025-b8e1-09d36e331662" (UID: "786eb420-ea6c-4025-b8e1-09d36e331662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:35.903806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:35.903786    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz" pod "786eb420-ea6c-4025-b8e1-09d36e331662" (UID: "786eb420-ea6c-4025-b8e1-09d36e331662"). InnerVolumeSpecName "kube-api-access-tp8hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:36.003329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.003268    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786eb420-ea6c-4025-b8e1-09d36e331662-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:36.003329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.003327    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp8hz\" (UniqueName: \"kubernetes.io/projected/786eb420-ea6c-4025-b8e1-09d36e331662-kube-api-access-tp8hz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:36.367253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.367216    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:36.499107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.499058    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:36.687920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.687888    6738 generic.go:356] "Generic (PLEG): container finished" podID="4482d977-9965-4541-9ccd-2535df24055b" containerID="96384e511c07e312caff4aab5f1519db08771a287ae0f4900053eed7dc98458b" exitCode=0
Oct 30 07:38:36.689984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.689951    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr" event={"ID":"4482d977-9965-4541-9ccd-2535df24055b","Type":"ContainerDied","Data":"96384e511c07e312caff4aab5f1519db08771a287ae0f4900053eed7dc98458b"}
Oct 30 07:38:36.693682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.692728    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd" event={"ID":"786eb420-ea6c-4025-b8e1-09d36e331662","Type":"ContainerDied","Data":"86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3"}
Oct 30 07:38:36.694110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.694082    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86b42def98012e65ae2fc7f5440ed11ea8fe88a9b68590d91cda83d4de4c52b3"
Oct 30 07:38:36.694278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:36.693167    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"
Oct 30 07:38:36.846363 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:36.839218    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-00a7322d0aa7: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:36.846363 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:36.839248    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:36.846363 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:36.839336    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials podName:85ff34c8-f82a-4e47-874b-f969f0d63dff nodeName:}" failed. No retries permitted until 2026-10-30 07:38:37.339314677 +0000 UTC m=+2477.772078562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff") : secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:37.346706 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:37.346669    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-00a7322d0aa7: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:37.346706 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:37.346704    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:37.346995 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:37.346790    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials podName:85ff34c8-f82a-4e47-874b-f969f0d63dff nodeName:}" failed. No retries permitted until 2026-10-30 07:38:38.346767825 +0000 UTC m=+2478.779531696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff") : secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:37.521760 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:37.521722    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:38:37.641615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:37.641576    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:38.005850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.005823    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:38.076567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.076529    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:38.175872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.175828    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\" (UniqueName: \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\") pod \"4482d977-9965-4541-9ccd-2535df24055b\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") "
Oct 30 07:38:38.176268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.176245    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\") pod \"4482d977-9965-4541-9ccd-2535df24055b\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") "
Oct 30 07:38:38.176817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.176792    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\") pod \"4482d977-9965-4541-9ccd-2535df24055b\" (UID: \"4482d977-9965-4541-9ccd-2535df24055b\") "
Oct 30 07:38:38.187560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.187525    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca" pod "4482d977-9965-4541-9ccd-2535df24055b" (UID: "4482d977-9965-4541-9ccd-2535df24055b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:38.195327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.194416    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca" pod "4482d977-9965-4541-9ccd-2535df24055b" (UID: "4482d977-9965-4541-9ccd-2535df24055b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:38:38.207545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.207509    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr" pod "4482d977-9965-4541-9ccd-2535df24055b" (UID: "4482d977-9965-4541-9ccd-2535df24055b"). InnerVolumeSpecName "kube-api-access-kqxrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:38.280098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.280052    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4482d977-9965-4541-9ccd-2535df24055b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.280394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.280375    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxrr\" (UniqueName: \"kubernetes.io/projected/4482d977-9965-4541-9ccd-2535df24055b-kube-api-access-kqxrr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.280517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.280503    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4482d977-9965-4541-9ccd-2535df24055b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.382121 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:38.382085    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-00a7322d0aa7: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:38.382121 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:38.382127    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:38.382440 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:38.382196    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials podName:85ff34c8-f82a-4e47-874b-f969f0d63dff nodeName:}" failed. No retries permitted until 2026-10-30 07:38:40.382176752 +0000 UTC m=+2480.814940637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff") : secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:38.515488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.515396    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" podUID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" containerName="payload-processing" containerID="cri-o://243ea894062da895260a22a403a7d9b66cb0ff154bec110b009253ca6709dc49" gracePeriod=25
Oct 30 07:38:38.721879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.721846    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"
Oct 30 07:38:38.721879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.721860    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr" event={"ID":"4482d977-9965-4541-9ccd-2535df24055b","Type":"ContainerDied","Data":"77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71"}
Oct 30 07:38:38.722116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.721919    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f0359f1a853480a875a887e4845fa7f0b74dba38e64c8db86f475c4f675d71"
Oct 30 07:38:38.723985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.723946    6738 generic.go:356] "Generic (PLEG): container finished" podID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" containerID="243ea894062da895260a22a403a7d9b66cb0ff154bec110b009253ca6709dc49" exitCode=0
Oct 30 07:38:38.724804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.724034    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" event={"ID":"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5","Type":"ContainerDied","Data":"243ea894062da895260a22a403a7d9b66cb0ff154bec110b009253ca6709dc49"}
Oct 30 07:38:38.789911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.789882    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:38:38.888146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\") pod \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") "
Oct 30 07:38:38.888352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888200    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\") pod \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") "
Oct 30 07:38:38.888433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888423    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\") pod \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") "
Oct 30 07:38:38.888487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888471    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\") pod \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\" (UID: \"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5\") "
Oct 30 07:38:38.888589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888563    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume" pod "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" (UID: "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:38.888864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.888833    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume" pod "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" (UID: "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:38.889151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.889129    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.889258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.889153    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.890814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.890789    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials" pod "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" (UID: "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:38.890911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.890823    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9" pod "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" (UID: "b84bc3fe-0477-4013-a4e6-5bd5190e9ff5"). InnerVolumeSpecName "kube-api-access-kzcm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:38.990705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.990667    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzcm9\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-kube-api-access-kzcm9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:38.990705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:38.990700    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:39.786536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:39.786503    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"
Oct 30 07:38:39.786992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:39.786524    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n" event={"ID":"b84bc3fe-0477-4013-a4e6-5bd5190e9ff5","Type":"ContainerDied","Data":"8791b3d90a3588dd3db6632690024d19ca4fe99312bbad79ec96e352692eb718"}
Oct 30 07:38:39.786992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:39.786604    6738 scope.go:122] "RemoveContainer" containerID="243ea894062da895260a22a403a7d9b66cb0ff154bec110b009253ca6709dc49"
Oct 30 07:38:39.891777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:39.891744    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:38:39.903160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:39.903126    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-c5f4zxs7n"]
Oct 30 07:38:40.104598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:40.098723    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:40.412804 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:40.412768    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-00a7322d0aa7: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:40.412804 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:40.412800    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:40.413033 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:40.412882    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials podName:85ff34c8-f82a-4e47-874b-f969f0d63dff nodeName:}" failed. No retries permitted until 2026-10-30 07:38:44.412867726 +0000 UTC m=+2484.845631620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff") : secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:41.694190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:41.694148    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" path="/var/lib/kubelet/pods/b84bc3fe-0477-4013-a4e6-5bd5190e9ff5/volumes"
Oct 30 07:38:42.066497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:42.066391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:43.804921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:43.804883    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:38:43.818761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:43.818726    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:43.942162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:43.942107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:44.192316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:44.188417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:44.483042 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:44.482961    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-00a7322d0aa7: secret "agc-cred-00a7322d0aa7" not found
Oct 30 07:38:44.483042 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:44.482990    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1f700/agc-cred-2c0ab0fb80d7: secret "agc-cred-2c0ab0fb80d7" not found
Oct 30 07:38:44.483042 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:44.483001    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8: [secret "agc-cred-00a7322d0aa7" not found, secret "agc-cred-2c0ab0fb80d7" not found]
Oct 30 07:38:44.483284 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:44.483058    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials podName:85ff34c8-f82a-4e47-874b-f969f0d63dff nodeName:}" failed. No retries permitted until 2026-10-30 07:38:52.483041013 +0000 UTC m=+2492.915804884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff") : [secret "agc-cred-00a7322d0aa7" not found, secret "agc-cred-2c0ab0fb80d7" not found]
Oct 30 07:38:44.528350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:44.528310    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:45.685163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:45.685125    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:45.807282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:45.807250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:45.918608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:45.918568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:46.330921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:46.330882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:46.443739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:46.443699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:48.145085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:48.145050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:48.806038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:48.805986    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" podUID="85ff34c8-f82a-4e47-874b-f969f0d63dff" containerName="payload-processing" containerID="cri-o://9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a" gracePeriod=25
Oct 30 07:38:49.088170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.088144    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:38:49.160848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.160803    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\") pod \"85ff34c8-f82a-4e47-874b-f969f0d63dff\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") "
Oct 30 07:38:49.161352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.160974    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\") pod \"85ff34c8-f82a-4e47-874b-f969f0d63dff\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") "
Oct 30 07:38:49.161352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.161008    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\") pod \"85ff34c8-f82a-4e47-874b-f969f0d63dff\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") "
Oct 30 07:38:49.161352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.161262    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\") pod \"85ff34c8-f82a-4e47-874b-f969f0d63dff\" (UID: \"85ff34c8-f82a-4e47-874b-f969f0d63dff\") "
Oct 30 07:38:49.161553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.161462    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume" pod "85ff34c8-f82a-4e47-874b-f969f0d63dff" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:49.161553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.161528    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume" pod "85ff34c8-f82a-4e47-874b-f969f0d63dff" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:49.162095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.162068    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:49.162337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.162270    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ff34c8-f82a-4e47-874b-f969f0d63dff-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:49.163398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.163373    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9" pod "85ff34c8-f82a-4e47-874b-f969f0d63dff" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff"). InnerVolumeSpecName "kube-api-access-h2ht9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:49.164131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.164104    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials" pod "85ff34c8-f82a-4e47-874b-f969f0d63dff" (UID: "85ff34c8-f82a-4e47-874b-f969f0d63dff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:49.263761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.263719    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:49.263761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.263755    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ht9\" (UniqueName: \"kubernetes.io/projected/85ff34c8-f82a-4e47-874b-f969f0d63dff-kube-api-access-h2ht9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:38:49.710533 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:49.710500    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ff34c8_f82a_4e47_874b_f969f0d63dff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 07:38:49.868781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.868744    6738 generic.go:356] "Generic (PLEG): container finished" podID="85ff34c8-f82a-4e47-874b-f969f0d63dff" containerID="9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a" exitCode=0
Oct 30 07:38:49.868950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.868809    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"
Oct 30 07:38:49.868950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.868821    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" event={"ID":"85ff34c8-f82a-4e47-874b-f969f0d63dff","Type":"ContainerDied","Data":"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"}
Oct 30 07:38:49.868950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.868881    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8" event={"ID":"85ff34c8-f82a-4e47-874b-f969f0d63dff","Type":"ContainerDied","Data":"2bae3bb7f1d679343175690607b587eb2cca28c26f3b5f6de3f061a2a5919143"}
Oct 30 07:38:49.868950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.868911    6738 scope.go:122] "RemoveContainer" containerID="9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"
Oct 30 07:38:49.880334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.880289    6738 scope.go:122] "RemoveContainer" containerID="9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"
Oct 30 07:38:49.880677 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:38:49.880654    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a\": container with ID starting with 9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a not found: ID does not exist" containerID="9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"
Oct 30 07:38:49.880752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.880686    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a"} err="failed to get container status \"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a\": rpc error: code = NotFound desc = could not find container \"9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a\": container with ID starting with 9e79368ccb30c21f2232169efe20e2faec03eca26f81037747b68495ed502b5a not found: ID does not exist"
Oct 30 07:38:49.924889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.924854    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:38:49.935676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:49.935646    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-58dblhkm8"]
Oct 30 07:38:50.038470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:50.038380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:50.912109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:50.912068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:51.059265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:51.059230    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:51.693236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:51.693204    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ff34c8-f82a-4e47-874b-f969f0d63dff" path="/var/lib/kubelet/pods/85ff34c8-f82a-4e47-874b-f969f0d63dff/volumes"
Oct 30 07:38:51.956182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:51.956092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:52.787571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:52.787533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:52.910733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:52.910698    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:53.060326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:53.060231    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:53.868017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:53.867983    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:55.707265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:55.707228    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:57.366032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:57.365996    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:57.616984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:57.616869    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:57.767319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:57.755359    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:57.884391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:57.883161    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:58.177212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:58.177128    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:58.303779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:58.303744    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:58.465172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:58.465080    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:59.036609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:59.036573    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:59.160051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:59.160007    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:59.846440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:59.846407    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:38:59.958144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:38:59.958110    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:00.075502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:00.075470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:00.372805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:00.372771    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:00.560086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:00.530909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:00.720970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:00.720873    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:01.452430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.452384    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:01.728566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.728484    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:39:01.729474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729451    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4482d977-9965-4541-9ccd-2535df24055b" containerName="revoke-keys"
Oct 30 07:39:01.729474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729473    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4482d977-9965-4541-9ccd-2535df24055b" containerName="revoke-keys"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729523    6738 cpu_manager.go:414] "containerMap: removing container" podUID="786eb420-ea6c-4025-b8e1-09d36e331662" containerName="revoke-keys"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729530    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786eb420-ea6c-4025-b8e1-09d36e331662" containerName="revoke-keys"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729564    6738 cpu_manager.go:414] "containerMap: removing container" podUID="85ff34c8-f82a-4e47-874b-f969f0d63dff" containerName="payload-processing"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729573    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ff34c8-f82a-4e47-874b-f969f0d63dff" containerName="payload-processing"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729602    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" containerName="payload-processing"
Oct 30 07:39:01.729670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729611    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" containerName="payload-processing"
Oct 30 07:39:01.729899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729811    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4482d977-9965-4541-9ccd-2535df24055b" containerName="revoke-keys"
Oct 30 07:39:01.729899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729833    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="786eb420-ea6c-4025-b8e1-09d36e331662" containerName="revoke-keys"
Oct 30 07:39:01.729899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729842    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ff34c8-f82a-4e47-874b-f969f0d63dff" containerName="payload-processing"
Oct 30 07:39:01.729899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.729853    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84bc3fe-0477-4013-a4e6-5bd5190e9ff5" containerName="payload-processing"
Oct 30 07:39:01.735640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.735617    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.751510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.751480    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:39:01.767360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.758512    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8f1df\""
Oct 30 07:39:01.767360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.758512    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"openshift-service-ca.crt\""
Oct 30 07:39:01.767360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.758591    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"kube-root-ca.crt\""
Oct 30 07:39:01.767360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.759135    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"routing-overlay\""
Oct 30 07:39:01.790973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.790935    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xrg\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.791179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.791070    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.791179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.791127    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.791326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.791303    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.793923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.793892    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"e2e-cmp-legacy-fbf124-provider-cred\""
Oct 30 07:39:01.794037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.793927    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-e2e--97d1e837-dockercfg-7tjdf\""
Oct 30 07:39:01.892434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.892392    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.892662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.892601    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xrg\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.892742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.892671    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.892742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.892731    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.893471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.893434    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.893636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.893577    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.895190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.895162    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:01.918853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:01.918815    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xrg\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:02.054822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:02.054727    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:02.385051 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:02.385016    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662bea70_de4f_4773_8c3f_780af9a08526.slice/crio-8abe3da8d681b95c1741be1cf0eeb8558f0e3c4de4fa9136041ee4258872439e WatchSource:0}: Error finding container 8abe3da8d681b95c1741be1cf0eeb8558f0e3c4de4fa9136041ee4258872439e: Status 404 returned error can't find the container with id 8abe3da8d681b95c1741be1cf0eeb8558f0e3c4de4fa9136041ee4258872439e
Oct 30 07:39:02.385051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:02.385031    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:39:02.559062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:02.559023    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:03.041755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:03.041704    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" event={"ID":"662bea70-de4f-4773-8c3f-780af9a08526","Type":"ContainerStarted","Data":"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"}
Oct 30 07:39:03.041951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:03.041772    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" event={"ID":"662bea70-de4f-4773-8c3f-780af9a08526","Type":"ContainerStarted","Data":"8abe3da8d681b95c1741be1cf0eeb8558f0e3c4de4fa9136041ee4258872439e"}
Oct 30 07:39:03.041951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:03.041810    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:03.088805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:03.088734    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" podStartSLOduration=2.088696237 podStartE2EDuration="2.088696237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:03.083568204 +0000 UTC m=+2503.516332090" watchObservedRunningTime="2026-10-30 07:39:03.088696237 +0000 UTC m=+2503.521460148"
Oct 30 07:39:04.404336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:04.404286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:06.274092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:06.274055    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:07.521802 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:39:07.521765    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:39:08.042756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:08.042719    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:10.034318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:10.034199    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:11.851540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:11.851506    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:13.740256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:13.740214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:14.054097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:14.054013    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:39:15.531632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:15.531591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:17.739662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:17.739627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:19.740803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:19.740765    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:20.618115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.618077    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"]
Oct 30 07:39:20.622965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.622936    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.641122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.641089    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"]
Oct 30 07:39:20.648125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.648102    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:39:20.648416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.648059    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:39:20.660972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.660933    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlwz\" (UniqueName: \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.661162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.661109    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.661162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.661146    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.762990 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.762958    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlwz\" (UniqueName: \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.763579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.763106    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.763579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.763159    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.763862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.763837    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.765636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.765612    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.787432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.787402    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlwz\" (UniqueName: \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\") pod \"maas-api-revoke-sub-0ab411df-97q6t\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:20.941969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:20.941930    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:21.247782 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:21.247750    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1980d93_9c03_48f9_acbc_f1d36e3b8e3e.slice/crio-023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c WatchSource:0}: Error finding container 023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c: Status 404 returned error can't find the container with id 023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c
Oct 30 07:39:21.248855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:21.248769    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"]
Oct 30 07:39:21.647308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:21.647263    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:22.255283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:22.255246    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t" event={"ID":"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e","Type":"ContainerStarted","Data":"e769f9ef5e23aa5cbf2f93c1db86460e31db6de75a9a9f6bf360ba058e12b7ee"}
Oct 30 07:39:22.255697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:22.255312    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t" event={"ID":"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e","Type":"ContainerStarted","Data":"023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c"}
Oct 30 07:39:22.301099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:22.301038    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t" podStartSLOduration=2.301023907 podStartE2EDuration="2.301023907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:22.297577447 +0000 UTC m=+2522.730341339" watchObservedRunningTime="2026-10-30 07:39:22.301023907 +0000 UTC m=+2522.733787799"
Oct 30 07:39:23.264513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:23.264480    6738 generic.go:356] "Generic (PLEG): container finished" podID="b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" containerID="e769f9ef5e23aa5cbf2f93c1db86460e31db6de75a9a9f6bf360ba058e12b7ee" exitCode=0
Oct 30 07:39:23.264953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:23.264562    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t" event={"ID":"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e","Type":"ContainerDied","Data":"e769f9ef5e23aa5cbf2f93c1db86460e31db6de75a9a9f6bf360ba058e12b7ee"}
Oct 30 07:39:23.413569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:23.413535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:24.442095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.442069    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:24.515868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.515831    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\") pod \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") "
Oct 30 07:39:24.516078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.516005    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\" (UniqueName: \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\") pod \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") "
Oct 30 07:39:24.516167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.516156    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\") pod \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\" (UID: \"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e\") "
Oct 30 07:39:24.516277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.516250    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca" pod "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" (UID: "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:24.516671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.516650    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:24.518161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.518131    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz" pod "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" (UID: "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e"). InnerVolumeSpecName "kube-api-access-8xlwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:24.518252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.518192    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca" pod "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" (UID: "b1980d93-9c03-48f9-acbc-f1d36e3b8e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:39:24.618430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.618395    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlwz\" (UniqueName: \"kubernetes.io/projected/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-kube-api-access-8xlwz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:24.618430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:24.618425    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:25.248325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:25.248278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:25.282678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:25.282646    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"
Oct 30 07:39:25.282883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:25.282675    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t" event={"ID":"b1980d93-9c03-48f9-acbc-f1d36e3b8e3e","Type":"ContainerDied","Data":"023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c"}
Oct 30 07:39:25.282883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:25.282724    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="023430135f7c89cb03911c932050b8c90ca7980c08a154971f67a21276a51d6c"
Oct 30 07:39:27.217186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:27.217150    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:29.158847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:29.158749    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:29.283006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:29.282975    6738 scope.go:122] "RemoveContainer" containerID="fccbcc4c1b2eb7bc9d1f15c607a087b7b00bf548ea8c05474ed83b7d2e80cecf"
Oct 30 07:39:31.051171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:31.051134    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:31.907561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:31.907524    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:32.040066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:32.040029    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:32.758131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:32.758093    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:33.136817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:33.136767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:34.033367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:34.033333    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:34.152867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:34.152830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:35.327572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.327534    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:35.472265 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.472226    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:35.734477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.734445    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:35.735266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.735249    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" containerName="revoke-keys"
Oct 30 07:39:35.735363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.735267    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" containerName="revoke-keys"
Oct 30 07:39:35.735431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.735396    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" containerName="revoke-keys"
Oct 30 07:39:35.741078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.741058    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.759052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.759021    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:35.772342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.767017    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a3d385576abf\""
Oct 30 07:39:35.772342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.767082    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 07:39:35.772342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.767148    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 07:39:35.772342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.767147    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 07:39:35.775024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.774984    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 07:39:35.775213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.775080    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zrwr2\""
Oct 30 07:39:35.872965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.872917    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62lz\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.873780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.873750    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.874530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.874507    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.874777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.874759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.993355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.993201    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.993769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.993736    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62lz\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.994003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.993985    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.994195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.994180    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.998129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.997104    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.998129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.997198    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:35.998129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:35.998034    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:36.019786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:36.019747    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62lz\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\") pod \"payload-processing-external-model-675d78f79c-bs482\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:36.061524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:36.061487    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:36.377182 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:36.377150    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb084dc9f_399d_4fb9_a94c_94192617c80b.slice/crio-c170f2974407579fc79e0b6158afa75f6511ca0355333a04975edd48fcba41a0 WatchSource:0}: Error finding container c170f2974407579fc79e0b6158afa75f6511ca0355333a04975edd48fcba41a0: Status 404 returned error can't find the container with id c170f2974407579fc79e0b6158afa75f6511ca0355333a04975edd48fcba41a0
Oct 30 07:39:36.377638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:36.377461    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:36.443005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:36.442978    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" event={"ID":"b084dc9f-399d-4fb9-a94c-94192617c80b","Type":"ContainerStarted","Data":"c170f2974407579fc79e0b6158afa75f6511ca0355333a04975edd48fcba41a0"}
Oct 30 07:39:37.254025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:37.253986    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:37.453106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:37.453062    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" event={"ID":"b084dc9f-399d-4fb9-a94c-94192617c80b","Type":"ContainerStarted","Data":"cc46b5459980f62d689cefb19e8572e11115f3a839088043931a4bde8bdd75a4"}
Oct 30 07:39:37.453675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:37.453309    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:37.499829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:37.499759    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" podStartSLOduration=2.499746336 podStartE2EDuration="2.499746336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:37.497041769 +0000 UTC m=+2537.929805666" watchObservedRunningTime="2026-10-30 07:39:37.499746336 +0000 UTC m=+2537.932510229"
Oct 30 07:39:37.521052 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:39:37.520967    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:39:39.049631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:39.049595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:40.217162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:40.217130    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:40.234947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:40.234913    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:40.365647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:40.365606    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:41.126044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:41.126002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:41.390447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:41.390352    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:41.503615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:41.503573    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:42.445012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:42.444977    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:43.351055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.351022    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"]
Oct 30 07:39:43.356103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.356080    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.367799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.367769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:43.379079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.379045    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"]
Oct 30 07:39:43.442217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442168    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442232    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4btt\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-kube-api-access-q4btt\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442438    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442473    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442501    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442629    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442690    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442801    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384cf69e-4f91-4b58-8001-875083896c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.442864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.442850    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384cf69e-4f91-4b58-8001-875083896c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.543990    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544042    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544104    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544126    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544219    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384cf69e-4f91-4b58-8001-875083896c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544267    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384cf69e-4f91-4b58-8001-875083896c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544422    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544470    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4btt\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-kube-api-access-q4btt\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544506    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544544    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.544592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544565    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.545196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.544839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.545196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.545011    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384cf69e-4f91-4b58-8001-875083896c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.546619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.546593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384cf69e-4f91-4b58-8001-875083896c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.546920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.546899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384cf69e-4f91-4b58-8001-875083896c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.568366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.568336    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.568573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.568551    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4btt\" (UniqueName: \"kubernetes.io/projected/384cf69e-4f91-4b58-8001-875083896c42-kube-api-access-q4btt\") pod \"maas-default-gateway-openshift-default-6dcf757899-7f465\" (UID: \"384cf69e-4f91-4b58-8001-875083896c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:43.677278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:43.677236    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:44.006125 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:44.006087    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384cf69e_4f91_4b58_8001_875083896c42.slice/crio-7d8f619fd04e4a2b7557d0ff9985df72b62a5d3c92d29656c34335b6ee64abff WatchSource:0}: Error finding container 7d8f619fd04e4a2b7557d0ff9985df72b62a5d3c92d29656c34335b6ee64abff: Status 404 returned error can't find the container with id 7d8f619fd04e4a2b7557d0ff9985df72b62a5d3c92d29656c34335b6ee64abff
Oct 30 07:39:44.006472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.006447    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"]
Oct 30 07:39:44.009520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.009485    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:44.009597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.009564    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:44.009652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.009604    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:44.517373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.517340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" event={"ID":"384cf69e-4f91-4b58-8001-875083896c42","Type":"ContainerStarted","Data":"a21865da058520d7c896e98b5e068941074eb3fdf215fe03d95c495fcc7048c5"}
Oct 30 07:39:44.517577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.517396    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" event={"ID":"384cf69e-4f91-4b58-8001-875083896c42","Type":"ContainerStarted","Data":"7d8f619fd04e4a2b7557d0ff9985df72b62a5d3c92d29656c34335b6ee64abff"}
Oct 30 07:39:44.567950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.567899    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podStartSLOduration=1.567883586 podStartE2EDuration="1.567883586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:44.563825703 +0000 UTC m=+2544.996589596" watchObservedRunningTime="2026-10-30 07:39:44.567883586 +0000 UTC m=+2545.000647478"
Oct 30 07:39:44.678358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:44.678319    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:45.193074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.193035    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:45.217960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.217917    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" podUID="b084dc9f-399d-4fb9-a94c-94192617c80b" containerName="payload-processing" containerID="cri-o://cc46b5459980f62d689cefb19e8572e11115f3a839088043931a4bde8bdd75a4" gracePeriod=25
Oct 30 07:39:45.528773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.528668    6738 generic.go:356] "Generic (PLEG): container finished" podID="b084dc9f-399d-4fb9-a94c-94192617c80b" containerID="cc46b5459980f62d689cefb19e8572e11115f3a839088043931a4bde8bdd75a4" exitCode=0
Oct 30 07:39:45.528773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.528753    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" event={"ID":"b084dc9f-399d-4fb9-a94c-94192617c80b","Type":"ContainerDied","Data":"cc46b5459980f62d689cefb19e8572e11115f3a839088043931a4bde8bdd75a4"}
Oct 30 07:39:45.595379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.595348    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:45.696430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.695924    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:45.696430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.695986    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:45.700476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.700437    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\") pod \"b084dc9f-399d-4fb9-a94c-94192617c80b\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") "
Oct 30 07:39:45.700640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.700596    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\") pod \"b084dc9f-399d-4fb9-a94c-94192617c80b\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") "
Oct 30 07:39:45.700755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.700655    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\") pod \"b084dc9f-399d-4fb9-a94c-94192617c80b\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") "
Oct 30 07:39:45.700826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.700764    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\") pod \"b084dc9f-399d-4fb9-a94c-94192617c80b\" (UID: \"b084dc9f-399d-4fb9-a94c-94192617c80b\") "
Oct 30 07:39:45.702241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.702204    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume" pod "b084dc9f-399d-4fb9-a94c-94192617c80b" (UID: "b084dc9f-399d-4fb9-a94c-94192617c80b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:45.702696 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.702667    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume" pod "b084dc9f-399d-4fb9-a94c-94192617c80b" (UID: "b084dc9f-399d-4fb9-a94c-94192617c80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:45.722648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.722609    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz" pod "b084dc9f-399d-4fb9-a94c-94192617c80b" (UID: "b084dc9f-399d-4fb9-a94c-94192617c80b"). InnerVolumeSpecName "kube-api-access-k62lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:45.729736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.729705    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials" pod "b084dc9f-399d-4fb9-a94c-94192617c80b" (UID: "b084dc9f-399d-4fb9-a94c-94192617c80b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:45.803674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.803583    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:45.803674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.803621    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b084dc9f-399d-4fb9-a94c-94192617c80b-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:45.803674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.803637    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:45.803674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:45.803651    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62lz\" (UniqueName: \"kubernetes.io/projected/b084dc9f-399d-4fb9-a94c-94192617c80b-kube-api-access-k62lz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:39:46.538037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.538007    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482"
Oct 30 07:39:46.538037 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.538020    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-bs482" event={"ID":"b084dc9f-399d-4fb9-a94c-94192617c80b","Type":"ContainerDied","Data":"c170f2974407579fc79e0b6158afa75f6511ca0355333a04975edd48fcba41a0"}
Oct 30 07:39:46.538359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.538097    6738 scope.go:122] "RemoveContainer" containerID="cc46b5459980f62d689cefb19e8572e11115f3a839088043931a4bde8bdd75a4"
Oct 30 07:39:46.618212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.618174    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:46.630260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.630228    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-bs482"]
Oct 30 07:39:46.690831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.690796    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:46.691111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:46.691084    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:47.299551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:47.299513    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:47.688707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:47.688667    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:47.690394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:47.690347    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:47.708747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:47.708698    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b084dc9f-399d-4fb9-a94c-94192617c80b" path="/var/lib/kubelet/pods/b084dc9f-399d-4fb9-a94c-94192617c80b/volumes"
Oct 30 07:39:47.946858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:47.946631    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:48.062024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:48.061990    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:48.679785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:48.679741    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:48.681617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:48.681580    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:49.565063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:49.565024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:49.680317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:49.678674    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:49.680317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:49.678737    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:49.838856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:49.838763    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:49.968747 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:49.968688    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:50.678485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:50.678439    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:50.679111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:50.678500    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:51.420167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:51.420043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:51.678131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:51.678031    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Oct 30 07:39:51.678323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:51.678118    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Oct 30 07:39:52.060870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:52.060774    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:53.678335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:53.678269    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:39:53.678988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:53.678943    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:39:53.754640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:53.754601    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:53.755158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:53.755126    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:53.756182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:53.756156    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465"
Oct 30 07:39:54.168609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:54.168572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:56.038974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.038933    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"]
Oct 30 07:39:56.039811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.039790    6738 cpu_manager.go:414] "containerMap: removing container" podUID="b084dc9f-399d-4fb9-a94c-94192617c80b" containerName="payload-processing"
Oct 30 07:39:56.039811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.039811    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b084dc9f-399d-4fb9-a94c-94192617c80b" containerName="payload-processing"
Oct 30 07:39:56.039995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.039954    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="b084dc9f-399d-4fb9-a94c-94192617c80b" containerName="payload-processing"
Oct 30 07:39:56.045230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.045199    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.062910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.062470    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzms\" (UniqueName: \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.062910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.062827    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.062910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.062861    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.063218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.063143    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"]
Oct 30 07:39:56.068586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.068548    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:39:56.068808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.068717    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:39:56.164195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.164150    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.164195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.164194    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.164523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.164248    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzms\" (UniqueName: \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.165034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.164981    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.166830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.166801    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.192277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.192240    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzms\" (UniqueName: \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\") pod \"maas-api-revoke-sub-155226b9-2tvhm\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.268829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.268794    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:56.363132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.363039    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:39:56.390205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.390166    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:56.829382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:56.787563    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"]
Oct 30 07:39:56.829382 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:56.825157    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e853cf2_86a3_40b7_b5ad_5fc1c89f71f7.slice/crio-838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274 WatchSource:0}: Error finding container 838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274: Status 404 returned error can't find the container with id 838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274
Oct 30 07:39:57.741078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:57.741036    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm" event={"ID":"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7","Type":"ContainerStarted","Data":"31004a35b693c85c0b07a9283c4db88fb35b2c4a5f2265a84af24634e6182dff"}
Oct 30 07:39:57.741511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:57.741108    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm" event={"ID":"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7","Type":"ContainerStarted","Data":"838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274"}
Oct 30 07:39:57.788463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:57.788397    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm" podStartSLOduration=1.788377085 podStartE2EDuration="1.788377085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:57.782425021 +0000 UTC m=+2558.215188924" watchObservedRunningTime="2026-10-30 07:39:57.788377085 +0000 UTC m=+2558.221140978"
Oct 30 07:39:58.377119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.377082    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:58.393575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.393540    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:39:58.398546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.398513    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.519949    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520020    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520152    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520179    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gmw\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520212    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520273    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520341    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520443    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.520926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.520506    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.525095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.524767    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:39:58.621773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.621734    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.621773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.621779    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.621839    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.621915    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.621951    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622069    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622126    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622247    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622272    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gmw\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622434    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622514    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622604    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.622755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.622682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.623420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.623283    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.626336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.626308    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.626485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.626463    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.646056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.645981    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.646056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.646035    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gmw\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-w479t\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:58.751420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.751382    6738 generic.go:356] "Generic (PLEG): container finished" podID="9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" containerID="31004a35b693c85c0b07a9283c4db88fb35b2c4a5f2265a84af24634e6182dff" exitCode=0
Oct 30 07:39:58.751864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.751455    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm" event={"ID":"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7","Type":"ContainerDied","Data":"31004a35b693c85c0b07a9283c4db88fb35b2c4a5f2265a84af24634e6182dff"}
Oct 30 07:39:58.826597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:58.826565    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:59.097699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.097660    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:59.166242 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:39:59.166210    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf68a9d_ba67_473e_a797_071eb232a1db.slice/crio-c4e829f995262661591827f23b94a4bb9bc6025bd724d9ebc31c8b506ac885f9 WatchSource:0}: Error finding container c4e829f995262661591827f23b94a4bb9bc6025bd724d9ebc31c8b506ac885f9: Status 404 returned error can't find the container with id c4e829f995262661591827f23b94a4bb9bc6025bd724d9ebc31c8b506ac885f9
Oct 30 07:39:59.166976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.166951    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:39:59.169853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.169820    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:59.169948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.169889    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:59.169948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.169920    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:39:59.231666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.231626    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:59.438079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.437993    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:59.733221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.733135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:39:59.789003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.788975    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" event={"ID":"1bf68a9d-ba67-473e-a797-071eb232a1db","Type":"ContainerStarted","Data":"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"}
Oct 30 07:39:59.789636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.789611    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" event={"ID":"1bf68a9d-ba67-473e-a797-071eb232a1db","Type":"ContainerStarted","Data":"c4e829f995262661591827f23b94a4bb9bc6025bd724d9ebc31c8b506ac885f9"}
Oct 30 07:39:59.828366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.826906    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:39:59.836745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:39:59.836668    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podStartSLOduration=1.836652832 podStartE2EDuration="1.836652832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:59.836084351 +0000 UTC m=+2560.268848258" watchObservedRunningTime="2026-10-30 07:39:59.836652832 +0000 UTC m=+2560.269416716"
Oct 30 07:40:00.034605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.034572    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:40:00.194506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.194467    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\" (UniqueName: \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\") pod \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") "
Oct 30 07:40:00.194805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.194784    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\") pod \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") "
Oct 30 07:40:00.194912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.194895    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\") pod \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\" (UID: \"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7\") "
Oct 30 07:40:00.201355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.196392    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca" pod "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" (UID: "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:00.201355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.198462    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca" pod "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" (UID: "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:40:00.201355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.199682    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms" pod "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" (UID: "9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7"). InnerVolumeSpecName "kube-api-access-sdzms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:00.297486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.297386    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:00.297486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.297428    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:00.297486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.297443    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzms\" (UniqueName: \"kubernetes.io/projected/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7-kube-api-access-sdzms\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:00.362340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.355547    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:00.495330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.494781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:00.893371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.889451    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:00.893371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.889513    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:00.954551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.950252    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"
Oct 30 07:40:00.965315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.963549    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm" event={"ID":"9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7","Type":"ContainerDied","Data":"838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274"}
Oct 30 07:40:00.965315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:00.963621    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838eea3d401bb0133ae29af7c3841f535ebbc5bbace5045a4462247187841274"
Oct 30 07:40:01.827598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:01.827557    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:01.827867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:01.827620    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:02.139792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:02.139743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:02.833329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:02.830138    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:02.833329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:02.830203    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:03.255868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:03.255832    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:03.392935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:03.392894    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:03.832318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:03.828988    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:03.832318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:03.829079    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:04.827638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:04.827601    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:04.828146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:04.827658    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:05.242380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:05.242343    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:05.828084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:05.828040    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:05.828546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:05.828116    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:06.828087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.828044    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-w479t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 07:40:06.828648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.828132    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 07:40:06.839143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.839107    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"]
Oct 30 07:40:06.840012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.839989    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" containerName="revoke-keys"
Oct 30 07:40:06.840012 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.840008    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" containerName="revoke-keys"
Oct 30 07:40:06.840238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.840177    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" containerName="revoke-keys"
Oct 30 07:40:06.844584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.844560    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:06.862146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.862113    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"]
Oct 30 07:40:06.868941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.868912    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:40:06.869173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.868932    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:40:06.934914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.934876    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:06.935124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.935057    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6dn\" (UniqueName: \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:06.935250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:06.935234    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.036695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.036658    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6dn\" (UniqueName: \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.036903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.036818    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.036903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.036867    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.037536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.037506    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.039324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.039284    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.061634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.061600    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6dn\" (UniqueName: \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\") pod \"maas-api-revoke-sub-0596e6fa-ggjft\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.151876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.151836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:07.165089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.165054    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:07.522867 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:07.522773    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:40:07.539499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:07.539466    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"]
Oct 30 07:40:07.540835 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:07.540800    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf892d204_36e4_4400_adb2_8a4befcc6754.slice/crio-f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af WatchSource:0}: Error finding container f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af: Status 404 returned error can't find the container with id f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af
Oct 30 07:40:08.050790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.050692    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft" event={"ID":"f892d204-36e4-4400-adb2-8a4befcc6754","Type":"ContainerStarted","Data":"4b7b205ec0abf8cb1a865ae40a6461e9368bd555e102d35486a282f12848af3d"}
Oct 30 07:40:08.050790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.050761    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft" event={"ID":"f892d204-36e4-4400-adb2-8a4befcc6754","Type":"ContainerStarted","Data":"f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af"}
Oct 30 07:40:08.096727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.096662    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft" podStartSLOduration=2.096648296 podStartE2EDuration="2.096648296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:08.092017188 +0000 UTC m=+2568.524781083" watchObservedRunningTime="2026-10-30 07:40:08.096648296 +0000 UTC m=+2568.529412189"
Oct 30 07:40:08.371768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.371632    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:40:08.372172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.372148    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:40:08.373047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:08.373023    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:40:09.037655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:09.037620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:09.060851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:09.060816    6738 generic.go:356] "Generic (PLEG): container finished" podID="f892d204-36e4-4400-adb2-8a4befcc6754" containerID="4b7b205ec0abf8cb1a865ae40a6461e9368bd555e102d35486a282f12848af3d" exitCode=0
Oct 30 07:40:09.061261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:09.060901    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft" event={"ID":"f892d204-36e4-4400-adb2-8a4befcc6754","Type":"ContainerDied","Data":"4b7b205ec0abf8cb1a865ae40a6461e9368bd555e102d35486a282f12848af3d"}
Oct 30 07:40:10.246066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.246031    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:10.352217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.352128    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\") pod \"f892d204-36e4-4400-adb2-8a4befcc6754\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") "
Oct 30 07:40:10.352217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.352178    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\" (UniqueName: \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\") pod \"f892d204-36e4-4400-adb2-8a4befcc6754\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") "
Oct 30 07:40:10.352515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.352260    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\") pod \"f892d204-36e4-4400-adb2-8a4befcc6754\" (UID: \"f892d204-36e4-4400-adb2-8a4befcc6754\") "
Oct 30 07:40:10.352591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.352565    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca" pod "f892d204-36e4-4400-adb2-8a4befcc6754" (UID: "f892d204-36e4-4400-adb2-8a4befcc6754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:10.352932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.352911    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:10.354537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.354505    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca" pod "f892d204-36e4-4400-adb2-8a4befcc6754" (UID: "f892d204-36e4-4400-adb2-8a4befcc6754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:40:10.354673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.354545    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn" pod "f892d204-36e4-4400-adb2-8a4befcc6754" (UID: "f892d204-36e4-4400-adb2-8a4befcc6754"). InnerVolumeSpecName "kube-api-access-ft6dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:10.454914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.454877    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f892d204-36e4-4400-adb2-8a4befcc6754-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:10.454914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.454912    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6dn\" (UniqueName: \"kubernetes.io/projected/f892d204-36e4-4400-adb2-8a4befcc6754-kube-api-access-ft6dn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:10.966450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:10.966417    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:11.079183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:11.079153    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"
Oct 30 07:40:11.079183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:11.079157    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft" event={"ID":"f892d204-36e4-4400-adb2-8a4befcc6754","Type":"ContainerDied","Data":"f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af"}
Oct 30 07:40:11.079448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:11.079201    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65c7e750355caaf38a6a8660f855d3e3a1015948a871416ac7ea809ccc4a4af"
Oct 30 07:40:12.837784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:12.837748    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:14.648531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:14.648489    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:16.943160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:16.943130    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:18.285675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:18.285634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:18.426319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:18.418663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:19.458831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:19.458791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:20.411409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:20.411353    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:20.496930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:20.496895    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:40:20.634675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:20.634634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:20.758625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:20.758361    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:21.207317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:21.195090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:21.864891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:21.864860    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:22.217264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.217227    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"]
Oct 30 07:40:22.218078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.218057    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f892d204-36e4-4400-adb2-8a4befcc6754" containerName="revoke-keys"
Oct 30 07:40:22.218078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.218078    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f892d204-36e4-4400-adb2-8a4befcc6754" containerName="revoke-keys"
Oct 30 07:40:22.218250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.218195    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f892d204-36e4-4400-adb2-8a4befcc6754" containerName="revoke-keys"
Oct 30 07:40:22.233281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.233252    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.243384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.243357    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"]
Oct 30 07:40:22.252271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.252246    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:40:22.252429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.252246    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:40:22.319773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.319738    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"]
Oct 30 07:40:22.324506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.324484    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.344785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.344750    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"]
Oct 30 07:40:22.355159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.355119    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.355377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.355263    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7xn\" (UniqueName: \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.355377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.355367    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.457344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457276    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdl8j\" (UniqueName: \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.457543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457401    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.457543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457453    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.457647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457580    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7xn\" (UniqueName: \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.457711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457670    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.457795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.457774    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.458654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.458626    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.460089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.460069    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.481740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.481679    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7xn\" (UniqueName: \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\") pod \"maas-api-revoke-sub-c1b99658-dqqdx\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.553194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.553157    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:22.559218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.559183    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.559407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.559336    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.559577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.559560    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdl8j\" (UniqueName: \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.559903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.559877    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.561699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.561679    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.583579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.583550    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdl8j\" (UniqueName: \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\") pod \"maas-api-revoke-sub-a28ce42e-rmczt\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.642969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.641691    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:22.862752 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:22.862717    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0916e46_f23a_455b_ac08_f90f271f7c09.slice/crio-67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b WatchSource:0}: Error finding container 67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b: Status 404 returned error can't find the container with id 67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b
Oct 30 07:40:22.863495 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.863467    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"]
Oct 30 07:40:22.964920 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:22.964887    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793decc0_4eb7_4bd6_a03a_81a97a2b2549.slice/crio-5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e WatchSource:0}: Error finding container 5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e: Status 404 returned error can't find the container with id 5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e
Oct 30 07:40:22.965378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:22.965357    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"]
Oct 30 07:40:23.360996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:23.360957    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx" event={"ID":"c0916e46-f23a-455b-ac08-f90f271f7c09","Type":"ContainerStarted","Data":"e0a472463c3a1416a22c7db6c35c239cab9e03f1a58cdb13888ffc187986773e"}
Oct 30 07:40:23.361189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:23.361061    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx" event={"ID":"c0916e46-f23a-455b-ac08-f90f271f7c09","Type":"ContainerStarted","Data":"67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b"}
Oct 30 07:40:23.362372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:23.362321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt" event={"ID":"793decc0-4eb7-4bd6-a03a-81a97a2b2549","Type":"ContainerStarted","Data":"5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e"}
Oct 30 07:40:23.411504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:23.411448    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx" podStartSLOduration=1.411434004 podStartE2EDuration="1.411434004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:23.408835978 +0000 UTC m=+2583.841599876" watchObservedRunningTime="2026-10-30 07:40:23.411434004 +0000 UTC m=+2583.844197945"
Oct 30 07:40:23.730520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:23.730480    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:24.383153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:24.383119    6738 generic.go:356] "Generic (PLEG): container finished" podID="c0916e46-f23a-455b-ac08-f90f271f7c09" containerID="e0a472463c3a1416a22c7db6c35c239cab9e03f1a58cdb13888ffc187986773e" exitCode=0
Oct 30 07:40:24.383608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:24.383202    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx" event={"ID":"c0916e46-f23a-455b-ac08-f90f271f7c09","Type":"ContainerDied","Data":"e0a472463c3a1416a22c7db6c35c239cab9e03f1a58cdb13888ffc187986773e"}
Oct 30 07:40:24.385278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:24.385250    6738 generic.go:356] "Generic (PLEG): container finished" podID="793decc0-4eb7-4bd6-a03a-81a97a2b2549" containerID="5d0e1dacf46e114c0692e849c16390e0596c6c89b2e28dde5f8cfcbb6bd39514" exitCode=0
Oct 30 07:40:24.385425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:24.385347    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt" event={"ID":"793decc0-4eb7-4bd6-a03a-81a97a2b2549","Type":"ContainerDied","Data":"5d0e1dacf46e114c0692e849c16390e0596c6c89b2e28dde5f8cfcbb6bd39514"}
Oct 30 07:40:25.501244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.501189    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" podUID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" containerName="payload-processing" containerID="cri-o://aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c" gracePeriod=25
Oct 30 07:40:25.556909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.556871    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:25.678063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.677146    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:25.681546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.681522    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:25.837098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.837062    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\") pod \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") "
Oct 30 07:40:25.837508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.837481    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca" pod "793decc0-4eb7-4bd6-a03a-81a97a2b2549" (UID: "793decc0-4eb7-4bd6-a03a-81a97a2b2549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:25.837678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.837284    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\") pod \"c0916e46-f23a-455b-ac08-f90f271f7c09\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") "
Oct 30 07:40:25.837958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.837799    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\") pod \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") "
Oct 30 07:40:25.837958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.837874    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\" (UniqueName: \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\") pod \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\" (UID: \"793decc0-4eb7-4bd6-a03a-81a97a2b2549\") "
Oct 30 07:40:25.838095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.838018    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\") pod \"c0916e46-f23a-455b-ac08-f90f271f7c09\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") "
Oct 30 07:40:25.838095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.838074    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\" (UniqueName: \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\") pod \"c0916e46-f23a-455b-ac08-f90f271f7c09\" (UID: \"c0916e46-f23a-455b-ac08-f90f271f7c09\") "
Oct 30 07:40:25.839069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.838843    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:25.840914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.840889    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca" pod "c0916e46-f23a-455b-ac08-f90f271f7c09" (UID: "c0916e46-f23a-455b-ac08-f90f271f7c09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:25.841898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.841432    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca" pod "c0916e46-f23a-455b-ac08-f90f271f7c09" (UID: "c0916e46-f23a-455b-ac08-f90f271f7c09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:40:25.843016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.842995    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j" pod "793decc0-4eb7-4bd6-a03a-81a97a2b2549" (UID: "793decc0-4eb7-4bd6-a03a-81a97a2b2549"). InnerVolumeSpecName "kube-api-access-rdl8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:25.843157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.843136    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca" pod "793decc0-4eb7-4bd6-a03a-81a97a2b2549" (UID: "793decc0-4eb7-4bd6-a03a-81a97a2b2549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:40:25.843536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.843515    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn" pod "c0916e46-f23a-455b-ac08-f90f271f7c09" (UID: "c0916e46-f23a-455b-ac08-f90f271f7c09"). InnerVolumeSpecName "kube-api-access-bg7xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:25.855868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.855835    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:40:25.940820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.940732    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdl8j\" (UniqueName: \"kubernetes.io/projected/793decc0-4eb7-4bd6-a03a-81a97a2b2549-kube-api-access-rdl8j\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:25.940820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.940774    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:25.940820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.940790    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7xn\" (UniqueName: \"kubernetes.io/projected/c0916e46-f23a-455b-ac08-f90f271f7c09-kube-api-access-bg7xn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:25.940820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.940803    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0916e46-f23a-455b-ac08-f90f271f7c09-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:25.940820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:25.940819    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793decc0-4eb7-4bd6-a03a-81a97a2b2549-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.042022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.041979    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\") pod \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") "
Oct 30 07:40:26.042266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042189    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\") pod \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") "
Oct 30 07:40:26.042266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042230    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\") pod \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") "
Oct 30 07:40:26.042417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042265    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\") pod \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\" (UID: \"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b\") "
Oct 30 07:40:26.042482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042420    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume" pod "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" (UID: "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:26.042597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042574    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume" pod "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" (UID: "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:26.042987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042965    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.043102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.042989    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.044564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.044543    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials" pod "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" (UID: "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:26.044639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.044607    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk" pod "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" (UID: "8b86b66d-e43b-48d3-9b0c-ed6cd77a856b"). InnerVolumeSpecName "kube-api-access-8ltwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:26.144886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.144845    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltwk\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-kube-api-access-8ltwk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.144886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.144878    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.402932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.402896    6738 generic.go:356] "Generic (PLEG): container finished" podID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" containerID="aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c" exitCode=0
Oct 30 07:40:26.403129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.402944    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"
Oct 30 07:40:26.403129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.402983    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" event={"ID":"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b","Type":"ContainerDied","Data":"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"}
Oct 30 07:40:26.403129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.403045    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n" event={"ID":"8b86b66d-e43b-48d3-9b0c-ed6cd77a856b","Type":"ContainerDied","Data":"e70f72b8017e4c933020549940f2de5ebf7d59a97058c2ca736f10c0e9908a76"}
Oct 30 07:40:26.403129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.403075    6738 scope.go:122] "RemoveContainer" containerID="aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"
Oct 30 07:40:26.404795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.404772    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt" event={"ID":"793decc0-4eb7-4bd6-a03a-81a97a2b2549","Type":"ContainerDied","Data":"5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e"}
Oct 30 07:40:26.404909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.404753    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"
Oct 30 07:40:26.404909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.404824    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5389f2703b484c44c2acfb493e3010929fb1484602d9345abd5916ff88567e4e"
Oct 30 07:40:26.406767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.406713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx" event={"ID":"c0916e46-f23a-455b-ac08-f90f271f7c09","Type":"ContainerDied","Data":"67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b"}
Oct 30 07:40:26.406767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.406736    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"
Oct 30 07:40:26.406767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.406750    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b43b68e7031e0de267aa01a1b62681011a5367038b7c98d74ac265f3d9de2b"
Oct 30 07:40:26.423522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.423478    6738 scope.go:122] "RemoveContainer" containerID="aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"
Oct 30 07:40:26.424198 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:26.424169    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c\": container with ID starting with aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c not found: ID does not exist" containerID="aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"
Oct 30 07:40:26.424404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.424219    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c"} err="failed to get container status \"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c\": rpc error: code = NotFound desc = could not find container \"aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c\": container with ID starting with aafbba90e758e015048fe7883d334b930bcc575c49348509c3173b324d989d0c not found: ID does not exist"
Oct 30 07:40:26.440865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.440833    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"]
Oct 30 07:40:26.441889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441864    6738 cpu_manager.go:414] "containerMap: removing container" podUID="793decc0-4eb7-4bd6-a03a-81a97a2b2549" containerName="revoke-keys"
Oct 30 07:40:26.441889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441887    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793decc0-4eb7-4bd6-a03a-81a97a2b2549" containerName="revoke-keys"
Oct 30 07:40:26.442089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441927    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" containerName="payload-processing"
Oct 30 07:40:26.442089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441935    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" containerName="payload-processing"
Oct 30 07:40:26.442089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441985    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c0916e46-f23a-455b-ac08-f90f271f7c09" containerName="revoke-keys"
Oct 30 07:40:26.442089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.441993    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0916e46-f23a-455b-ac08-f90f271f7c09" containerName="revoke-keys"
Oct 30 07:40:26.442312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.442199    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="793decc0-4eb7-4bd6-a03a-81a97a2b2549" containerName="revoke-keys"
Oct 30 07:40:26.442312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.442218    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" containerName="payload-processing"
Oct 30 07:40:26.442312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.442236    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0916e46-f23a-455b-ac08-f90f271f7c09" containerName="revoke-keys"
Oct 30 07:40:26.457224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.457191    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.459353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.459265    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"]
Oct 30 07:40:26.475483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.475106    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:40:26.475483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.475170    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:40:26.483146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.483114    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:40:26.487364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.487335    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-7d5frkf6n"]
Oct 30 07:40:26.651540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.651489    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.651540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.651541    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwtv\" (UniqueName: \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.652078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.651722    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.753770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.753677    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.753983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.753834    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwtv\" (UniqueName: \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.754027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.754016    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.754709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.754682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.756373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.756347    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.778170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.778137    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwtv\" (UniqueName: \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\") pod \"maas-api-revoke-sub-5f1d4092-7d25j\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:26.779841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:26.779819    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:27.090041 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:27.090011    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b1ff38_5680_4315_852c_c7a98c922b4b.slice/crio-cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935 WatchSource:0}: Error finding container cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935: Status 404 returned error can't find the container with id cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935
Oct 30 07:40:27.091354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.091327    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"]
Oct 30 07:40:27.418477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.418444    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j" event={"ID":"18b1ff38-5680-4315-852c-c7a98c922b4b","Type":"ContainerStarted","Data":"cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935"}
Oct 30 07:40:27.440953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.440916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:27.558361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.557183    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:27.669800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.669701    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:27.692641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:27.692608    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b86b66d-e43b-48d3-9b0c-ed6cd77a856b" path="/var/lib/kubelet/pods/8b86b66d-e43b-48d3-9b0c-ed6cd77a856b/volumes"
Oct 30 07:40:28.429382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:28.429344    6738 generic.go:356] "Generic (PLEG): container finished" podID="18b1ff38-5680-4315-852c-c7a98c922b4b" containerID="39e92e512f8fd618738b7520aae9b3db896bdd07490cc798e12b311a9975a20e" exitCode=0
Oct 30 07:40:28.429584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:28.429402    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j" event={"ID":"18b1ff38-5680-4315-852c-c7a98c922b4b","Type":"ContainerDied","Data":"39e92e512f8fd618738b7520aae9b3db896bdd07490cc798e12b311a9975a20e"}
Oct 30 07:40:29.271214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.271169    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:29.604457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.604430    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:29.695565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.695534    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\" (UniqueName: \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\") pod \"18b1ff38-5680-4315-852c-c7a98c922b4b\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") "
Oct 30 07:40:29.695758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.695580    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\") pod \"18b1ff38-5680-4315-852c-c7a98c922b4b\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") "
Oct 30 07:40:29.695758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.695664    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\") pod \"18b1ff38-5680-4315-852c-c7a98c922b4b\" (UID: \"18b1ff38-5680-4315-852c-c7a98c922b4b\") "
Oct 30 07:40:29.696388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.696355    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca" pod "18b1ff38-5680-4315-852c-c7a98c922b4b" (UID: "18b1ff38-5680-4315-852c-c7a98c922b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:29.698712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.698686    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca" pod "18b1ff38-5680-4315-852c-c7a98c922b4b" (UID: "18b1ff38-5680-4315-852c-c7a98c922b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:40:29.698877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.698853    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv" pod "18b1ff38-5680-4315-852c-c7a98c922b4b" (UID: "18b1ff38-5680-4315-852c-c7a98c922b4b"). InnerVolumeSpecName "kube-api-access-pkwtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:29.798274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.798237    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwtv\" (UniqueName: \"kubernetes.io/projected/18b1ff38-5680-4315-852c-c7a98c922b4b-kube-api-access-pkwtv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:29.798274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.798273    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:29.798538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:29.798287    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b1ff38-5680-4315-852c-c7a98c922b4b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:30.449772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:30.449742    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"
Oct 30 07:40:30.450182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:30.449774    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j" event={"ID":"18b1ff38-5680-4315-852c-c7a98c922b4b","Type":"ContainerDied","Data":"cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935"}
Oct 30 07:40:30.450182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:30.449833    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd9b1f28ca303360bf6d49a9be2a60cce383cf90634e80a210d2ad1f0dd7935"
Oct 30 07:40:31.064406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:31.064358    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:32.878363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:32.878288    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:33.109168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:33.109131    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:33.716143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:33.716109    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:34.048226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:34.048132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:34.161555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:34.161514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:34.506685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:34.506647    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:34.750259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:34.750219    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:34.876044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:34.876013    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.007433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.007393    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.134565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.134450    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.297757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.297707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.415531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.415434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.521984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.521944    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:35.523000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.522979    6738 cpu_manager.go:414] "containerMap: removing container" podUID="18b1ff38-5680-4315-852c-c7a98c922b4b" containerName="revoke-keys"
Oct 30 07:40:35.523081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.523003    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b1ff38-5680-4315-852c-c7a98c922b4b" containerName="revoke-keys"
Oct 30 07:40:35.523171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.523162    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b1ff38-5680-4315-852c-c7a98c922b4b" containerName="revoke-keys"
Oct 30 07:40:35.527524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.527502    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.546432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.546405    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:35.553585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.553555    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f5081\""
Oct 30 07:40:35.554419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.554392    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"routing-overlay\""
Oct 30 07:40:35.554559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.554473    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"openshift-service-ca.crt\""
Oct 30 07:40:35.554688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.554662    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"kube-root-ca.crt\""
Oct 30 07:40:35.559511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.559484    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"agc-cred-56d4535fe8f4\""
Oct 30 07:40:35.559638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.559545    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"payload-processing-external-model-e2e--8bb4c319-dockercfg-vmfs8\""
Oct 30 07:40:35.560448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.560421    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.560511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.560487    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.560565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.560525    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8jx\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.560604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.560564    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.662640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.662601    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.662827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.662653    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.662827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.662678    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8jx\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.662827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.662703    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.663515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.663487    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.663661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.663590    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.665240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.665216    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.685612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.685540    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:35.702845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.702809    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8jx\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:35.862479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:35.856468    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:36.025207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.022072    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:36.157562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.157481    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:36.275454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.275366    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:36.489739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.482032    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:36.511761 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:36.511693    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f07c7d5_7d5d_4541_b813_10d45dea0317.slice/crio-5d1075ccaabfcdba6c40b84e71df18a7d3897a6fe6c861b6ea001a0483023878 WatchSource:0}: Error finding container 5d1075ccaabfcdba6c40b84e71df18a7d3897a6fe6c861b6ea001a0483023878: Status 404 returned error can't find the container with id 5d1075ccaabfcdba6c40b84e71df18a7d3897a6fe6c861b6ea001a0483023878
Oct 30 07:40:36.631858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.631815    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" event={"ID":"7f07c7d5-7d5d-4541-b813-10d45dea0317","Type":"ContainerStarted","Data":"5d1075ccaabfcdba6c40b84e71df18a7d3897a6fe6c861b6ea001a0483023878"}
Oct 30 07:40:36.917366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:36.917331    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:37.051316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:37.050881    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:37.520664 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:37.520627    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:40:37.540886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:37.540852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:37.643166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:37.643126    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" event={"ID":"7f07c7d5-7d5d-4541-b813-10d45dea0317","Type":"ContainerStarted","Data":"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"}
Oct 30 07:40:37.643388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:37.643211    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:37.689957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:37.689901    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" podStartSLOduration=2.6898861849999998 podStartE2EDuration="2.689886185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:37.686660788 +0000 UTC m=+2598.119424682" watchObservedRunningTime="2026-10-30 07:40:37.689886185 +0000 UTC m=+2598.122650078"
Oct 30 07:40:37.789368 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:37.788435    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:37.789368 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:37.788468    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:37.789368 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:37.788529    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials podName:7f07c7d5-7d5d-4541-b813-10d45dea0317 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:38.28850672 +0000 UTC m=+2598.721270591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:38.151059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.151019    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:38.315274 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:38.301064    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:38.315274 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:38.301099    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:38.315274 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:38.301159    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials podName:7f07c7d5-7d5d-4541-b813-10d45dea0317 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:39.301140199 +0000 UTC m=+2599.733904092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:38.361663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.361622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:38.583725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.583635    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:38.610823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.610792    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:40:38.694277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.694222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:38.877494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:38.877433    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:39.334842 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:39.334754    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:39.334842 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:39.334783    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:39.334842 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:39.334838    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials podName:7f07c7d5-7d5d-4541-b813-10d45dea0317 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:41.334819385 +0000 UTC m=+2601.767583271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:39.510436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:39.510403    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:39.595389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:39.595289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:40.111197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:40.111159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:40.223347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:40.223278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:41.007196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:41.007157    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:41.132979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:41.132945    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:41.427317 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:41.427264    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:41.427317 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:41.427317    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:41.427615 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:41.427400    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials podName:7f07c7d5-7d5d-4541-b813-10d45dea0317 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:45.427364086 +0000 UTC m=+2605.860127963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:40:42.759230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:42.759176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:42.947787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:42.947749    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:43.255803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:43.255758    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:43.459831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:43.459797    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:43.598733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:43.598650    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:43.613316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:43.612145    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" podUID="662bea70-de4f-4773-8c3f-780af9a08526" containerName="payload-processing" containerID="cri-o://ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d" gracePeriod=25
Oct 30 07:40:44.150885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.150856    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:40:44.263886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.263848    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\") pod \"662bea70-de4f-4773-8c3f-780af9a08526\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") "
Oct 30 07:40:44.264111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.263921    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\") pod \"662bea70-de4f-4773-8c3f-780af9a08526\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") "
Oct 30 07:40:44.264111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264075    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\") pod \"662bea70-de4f-4773-8c3f-780af9a08526\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") "
Oct 30 07:40:44.264251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264107    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\") pod \"662bea70-de4f-4773-8c3f-780af9a08526\" (UID: \"662bea70-de4f-4773-8c3f-780af9a08526\") "
Oct 30 07:40:44.264371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume" pod "662bea70-de4f-4773-8c3f-780af9a08526" (UID: "662bea70-de4f-4773-8c3f-780af9a08526"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:44.264435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264363    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume" pod "662bea70-de4f-4773-8c3f-780af9a08526" (UID: "662bea70-de4f-4773-8c3f-780af9a08526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:44.264877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264850    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:44.264877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.264876    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662bea70-de4f-4773-8c3f-780af9a08526-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:44.266535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.266508    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials" pod "662bea70-de4f-4773-8c3f-780af9a08526" (UID: "662bea70-de4f-4773-8c3f-780af9a08526"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:44.267740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.267717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg" pod "662bea70-de4f-4773-8c3f-780af9a08526" (UID: "662bea70-de4f-4773-8c3f-780af9a08526"). InnerVolumeSpecName "kube-api-access-f8xrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:44.366809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.366722    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:44.366809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.366751    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8xrg\" (UniqueName: \"kubernetes.io/projected/662bea70-de4f-4773-8c3f-780af9a08526-kube-api-access-f8xrg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:44.673310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.673240    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" podUID="7f07c7d5-7d5d-4541-b813-10d45dea0317" containerName="payload-processing" containerID="cri-o://462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962" gracePeriod=25
Oct 30 07:40:44.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.861669    6738 generic.go:356] "Generic (PLEG): container finished" podID="662bea70-de4f-4773-8c3f-780af9a08526" containerID="ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d" exitCode=0
Oct 30 07:40:44.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.861707    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"
Oct 30 07:40:44.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.861747    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" event={"ID":"662bea70-de4f-4773-8c3f-780af9a08526","Type":"ContainerDied","Data":"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"}
Oct 30 07:40:44.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.861811    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" event={"ID":"662bea70-de4f-4773-8c3f-780af9a08526","Type":"ContainerDied","Data":"8abe3da8d681b95c1741be1cf0eeb8558f0e3c4de4fa9136041ee4258872439e"}
Oct 30 07:40:44.862315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.861842    6738 scope.go:122] "RemoveContainer" containerID="ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"
Oct 30 07:40:44.876268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.876246    6738 scope.go:122] "RemoveContainer" containerID="ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"
Oct 30 07:40:44.876735 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:44.876701    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d\": container with ID starting with ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d not found: ID does not exist" containerID="ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"
Oct 30 07:40:44.876840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.876746    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d"} err="failed to get container status \"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d\": rpc error: code = NotFound desc = could not find container \"ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d\": container with ID starting with ff4fe90ae7a67a43d58f2b30b5398f28b4686d399db7c9a050338cb26e8d874d not found: ID does not exist"
Oct 30 07:40:44.921339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.921287    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:40:44.942708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:44.942618    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f"]
Oct 30 07:40:45.081350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.064801    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-854646p8f" podUID="662bea70-de4f-4773-8c3f-780af9a08526" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.90:50052\" within 1s: context deadline exceeded"
Oct 30 07:40:45.081350 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:45.065228    6738 logging.go:55] [core] [Channel #9074 SubChannel #9075]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.90:50052", ServerName: "10.133.1.90:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.90:50052: operation was canceled"
Oct 30 07:40:45.081350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.065519    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:45.196368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.195848    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:45.268386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.268356    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:45.324733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.324692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:45.426717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.426679    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\") pod \"7f07c7d5-7d5d-4541-b813-10d45dea0317\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") "
Oct 30 07:40:45.426926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.426770    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\") pod \"7f07c7d5-7d5d-4541-b813-10d45dea0317\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") "
Oct 30 07:40:45.426926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.426805    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\") pod \"7f07c7d5-7d5d-4541-b813-10d45dea0317\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") "
Oct 30 07:40:45.426926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.426906    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\") pod \"7f07c7d5-7d5d-4541-b813-10d45dea0317\" (UID: \"7f07c7d5-7d5d-4541-b813-10d45dea0317\") "
Oct 30 07:40:45.427231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.427200    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume" pod "7f07c7d5-7d5d-4541-b813-10d45dea0317" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:45.427378 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.427226    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume" pod "7f07c7d5-7d5d-4541-b813-10d45dea0317" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:45.427461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.427444    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:45.427501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.427465    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f07c7d5-7d5d-4541-b813-10d45dea0317-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:45.429119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.429095    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx" pod "7f07c7d5-7d5d-4541-b813-10d45dea0317" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317"). InnerVolumeSpecName "kube-api-access-gj8jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:45.429274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.429144    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials" pod "7f07c7d5-7d5d-4541-b813-10d45dea0317" (UID: "7f07c7d5-7d5d-4541-b813-10d45dea0317"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:45.528419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.528330    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:45.528419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.528359    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8jx\" (UniqueName: \"kubernetes.io/projected/7f07c7d5-7d5d-4541-b813-10d45dea0317-kube-api-access-gj8jx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:40:45.692130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.692097    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:45.693661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.693637    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662bea70-de4f-4773-8c3f-780af9a08526" path="/var/lib/kubelet/pods/662bea70-de4f-4773-8c3f-780af9a08526/volumes"
Oct 30 07:40:45.871766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.871737    6738 generic.go:356] "Generic (PLEG): container finished" podID="7f07c7d5-7d5d-4541-b813-10d45dea0317" containerID="462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962" exitCode=0
Oct 30 07:40:45.871949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.871782    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"
Oct 30 07:40:45.871949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.871823    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" event={"ID":"7f07c7d5-7d5d-4541-b813-10d45dea0317","Type":"ContainerDied","Data":"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"}
Oct 30 07:40:45.871949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.871886    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv" event={"ID":"7f07c7d5-7d5d-4541-b813-10d45dea0317","Type":"ContainerDied","Data":"5d1075ccaabfcdba6c40b84e71df18a7d3897a6fe6c861b6ea001a0483023878"}
Oct 30 07:40:45.871949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.871915    6738 scope.go:122] "RemoveContainer" containerID="462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"
Oct 30 07:40:45.883839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.883817    6738 scope.go:122] "RemoveContainer" containerID="462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"
Oct 30 07:40:45.884179 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:40:45.884157    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962\": container with ID starting with 462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962 not found: ID does not exist" containerID="462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"
Oct 30 07:40:45.884255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.884193    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962"} err="failed to get container status \"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962\": rpc error: code = NotFound desc = could not find container \"462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962\": container with ID starting with 462c495858e69f0bc95e978ebd1acc4965984741bedaba1f6ae90c5407dd8962 not found: ID does not exist"
Oct 30 07:40:45.932331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.932278    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:45.944153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:45.944113    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f95xncv"]
Oct 30 07:40:47.119138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.119107    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:40:47.119978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.119958    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7f07c7d5-7d5d-4541-b813-10d45dea0317" containerName="payload-processing"
Oct 30 07:40:47.120036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.119979    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f07c7d5-7d5d-4541-b813-10d45dea0317" containerName="payload-processing"
Oct 30 07:40:47.120036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.120003    6738 cpu_manager.go:414] "containerMap: removing container" podUID="662bea70-de4f-4773-8c3f-780af9a08526" containerName="payload-processing"
Oct 30 07:40:47.120036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.120008    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662bea70-de4f-4773-8c3f-780af9a08526" containerName="payload-processing"
Oct 30 07:40:47.120140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.120130    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f07c7d5-7d5d-4541-b813-10d45dea0317" containerName="payload-processing"
Oct 30 07:40:47.120178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.120143    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="662bea70-de4f-4773-8c3f-780af9a08526" containerName="payload-processing"
Oct 30 07:40:47.125231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.125204    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.142134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.142102    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:40:47.149906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.149882    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"openshift-service-ca.crt\""
Oct 30 07:40:47.150069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.149881    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"agc-cred-f00723cbe17e\""
Oct 30 07:40:47.150069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.149882    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c1b5d7\""
Oct 30 07:40:47.150069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.149882    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"agc-cred-ad217a913a5d\""
Oct 30 07:40:47.150069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.150022    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-e2e--68f86622-dockercfg-7z9pp\""
Oct 30 07:40:47.152676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.152658    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"kube-root-ca.crt\""
Oct 30 07:40:47.152676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.152657    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"routing-overlay\""
Oct 30 07:40:47.163852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.163819    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7n8\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.164008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.163885    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.164076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.163998    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.164351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.164330    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.265266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.265217    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.265490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.265356    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7n8\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.265490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.265405    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.265490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.265452    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.266064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.266039    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.266064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.266053    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.268339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.268273    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.289444 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.289416    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7n8\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.442068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.442031    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.536050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.536015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:47.691947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.691914    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f07c7d5-7d5d-4541-b813-10d45dea0317" path="/var/lib/kubelet/pods/7f07c7d5-7d5d-4541-b813-10d45dea0317/volumes"
Oct 30 07:40:47.754533 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:47.754501    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543082cc_cf2a_47dc_9c1c_e8fa234ad3b7.slice/crio-b9b0c59b5bf25c699bd68be251ac26b0a78e532af1a238ca34f45db756e8a2ad WatchSource:0}: Error finding container b9b0c59b5bf25c699bd68be251ac26b0a78e532af1a238ca34f45db756e8a2ad: Status 404 returned error can't find the container with id b9b0c59b5bf25c699bd68be251ac26b0a78e532af1a238ca34f45db756e8a2ad
Oct 30 07:40:47.755150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.755121    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:40:47.904523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.904483    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" event={"ID":"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7","Type":"ContainerStarted","Data":"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"}
Oct 30 07:40:47.904716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.904583    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:47.904716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.904601    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" event={"ID":"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7","Type":"ContainerStarted","Data":"b9b0c59b5bf25c699bd68be251ac26b0a78e532af1a238ca34f45db756e8a2ad"}
Oct 30 07:40:47.947198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:47.947139    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" podStartSLOduration=0.947117533 podStartE2EDuration="947.117533ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:47.946729274 +0000 UTC m=+2608.379493167" watchObservedRunningTime="2026-10-30 07:40:47.947117533 +0000 UTC m=+2608.379881425"
Oct 30 07:40:48.007516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.007434    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:40:48.012673 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.012649    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.028474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028443    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-1e76919a7de9\""
Oct 30 07:40:48.028649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028515    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:40:48.028649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028517    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-6cd15efdbb3d\""
Oct 30 07:40:48.028649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028515    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:40:48.028649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028578    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-fpjr8\""
Oct 30 07:40:48.028890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.028667    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:40:48.029035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.029012    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:40:48.030114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.030096    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:40:48.090370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.090290    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.090565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.090476    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqbj\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.090565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.090522    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.090707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.090690    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.192705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.192669    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.193210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.192755    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqbj\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.193210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.192787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.193210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.192853    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.193562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.193539    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.193707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.193686    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.195240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.195217    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.216932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.216899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqbj\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.328420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.328328    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.641954 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:48.641922    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88dd0509_5d2d_40ea_bb97_8e940a462bc2.slice/crio-46d4fe2ef879e45aae629796b7b024250af151402cb28a4e8a27601516a2ef48 WatchSource:0}: Error finding container 46d4fe2ef879e45aae629796b7b024250af151402cb28a4e8a27601516a2ef48: Status 404 returned error can't find the container with id 46d4fe2ef879e45aae629796b7b024250af151402cb28a4e8a27601516a2ef48
Oct 30 07:40:48.643145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.643120    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:40:48.915083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.914976    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" event={"ID":"88dd0509-5d2d-40ea-bb97-8e940a462bc2","Type":"ContainerStarted","Data":"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"}
Oct 30 07:40:48.915083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.915043    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" event={"ID":"88dd0509-5d2d-40ea-bb97-8e940a462bc2","Type":"ContainerStarted","Data":"46d4fe2ef879e45aae629796b7b024250af151402cb28a4e8a27601516a2ef48"}
Oct 30 07:40:48.915083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.915071    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:40:48.960569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:48.960281    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" podStartSLOduration=1.96026444 podStartE2EDuration="1.96026444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:48.956257916 +0000 UTC m=+2609.389021810" watchObservedRunningTime="2026-10-30 07:40:48.96026444 +0000 UTC m=+2609.393028333"
Oct 30 07:40:49.563317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:49.548944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:50.148411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:50.148363    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:50.280424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:50.280378    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:51.181972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:51.181935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:51.313781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:51.313739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:51.499600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:51.499501    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:51.999365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:51.999331    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:52.134438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.134403    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:52.499385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.499339    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:52.683454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.683420    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:40:52.688235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.688213    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.706802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.706769    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:40:52.714868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.713452    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 07:40:52.714868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.713576    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8d2rr\""
Oct 30 07:40:52.714868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.713617    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c204db61a2ae\""
Oct 30 07:40:52.724351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.717258    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 07:40:52.724351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.717362    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 07:40:52.724351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.717432    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-031aedf881ae\""
Oct 30 07:40:52.724351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.717473    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 07:40:52.774574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.774475    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6v6t\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.774574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.774515    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.774819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.774733    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.774881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.774860    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.876640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.876608    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.876854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.876670    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.876903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.876866    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6v6t\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.876903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.876886    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.877546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.877514    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.877700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.877515    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.879578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.879558    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:52.901536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:52.901503    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6v6t\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\") pod \"payload-processing-external-model-6cc568d7d6-4dcvj\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") " pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:53.004379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.004341    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:53.008606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.008283    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:53.272232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.272198    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:53.579867 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:40:53.579826    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c7b92d_9336_4f09_ad90_cab59eaa55b2.slice/crio-f7c8c763b9ba37f23c8206f423adc49998d5d6dda9e59fc7d3aab844c61bb998 WatchSource:0}: Error finding container f7c8c763b9ba37f23c8206f423adc49998d5d6dda9e59fc7d3aab844c61bb998: Status 404 returned error can't find the container with id f7c8c763b9ba37f23c8206f423adc49998d5d6dda9e59fc7d3aab844c61bb998
Oct 30 07:40:53.581257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.581232    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:40:53.600251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.600222    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:53.964312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.964264    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" event={"ID":"09c7b92d-9336-4f09-ad90-cab59eaa55b2","Type":"ContainerStarted","Data":"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"}
Oct 30 07:40:53.964519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.964336    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" event={"ID":"09c7b92d-9336-4f09-ad90-cab59eaa55b2","Type":"ContainerStarted","Data":"f7c8c763b9ba37f23c8206f423adc49998d5d6dda9e59fc7d3aab844c61bb998"}
Oct 30 07:40:53.964519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:53.964367    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:40:54.009939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:54.009880    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" podStartSLOduration=2.009865036 podStartE2EDuration="2.009865036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:54.006149193 +0000 UTC m=+2614.438913087" watchObservedRunningTime="2026-10-30 07:40:54.009865036 +0000 UTC m=+2614.442628953"
Oct 30 07:40:55.578615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:55.578572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:57.477318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:57.469804    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:57.612857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:57.612696    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:58.224319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:58.223915    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:58.942319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:58.941480    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:40:59.564105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.563245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:59.678745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.678709    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:40:59.753021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.752983    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:40:59.759552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.759524    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.773153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.773115    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc84k\" (UniqueName: \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.773482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.773458    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.776862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.776836    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ad434e\""
Oct 30 07:40:59.781765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.781737    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:40:59.784260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.784225    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ad434e-dockercfg-9m5tl\""
Oct 30 07:40:59.884385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.883650    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc84k\" (UniqueName: \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.884385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.883744    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.917318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.884954    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.931328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.924207    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc84k\" (UniqueName: \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\") pod \"payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:40:59.931613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:40:59.931455    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:41:00.032326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.028415    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:41:00.067323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.043658    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.083467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.081678    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2bab4c-openshift-default-dockercfg-gqrfc\""
Oct 30 07:41:00.083644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.083536    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:41:00.109620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.109579    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110058    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110242    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzj6g\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110389    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110500    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110709    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.110873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.110858    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.111033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.111016    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.111215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.111201    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.177899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.177856    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.214702    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.214794    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.214861    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.214951    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.215004    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.215113    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.215174    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzj6g\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.215215    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.215313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.215245    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.216460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.216389    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.216460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.216397    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.216684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.216661    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.216935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.216912    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.217151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.217084    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.219117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.219070    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.221939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.221912    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.239053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.239022    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzj6g\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.240496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.240377    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.375006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.374968    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:00.482769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.482365    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:41:00.485875 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:41:00.485845    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d81899_c2b9_4536_9a5b_0202cc6dff08.slice/crio-008375da9d9ae377791b904e4e9bbf973b2dee9dd21815a36d59c625d1264b88 WatchSource:0}: Error finding container 008375da9d9ae377791b904e4e9bbf973b2dee9dd21815a36d59c625d1264b88: Status 404 returned error can't find the container with id 008375da9d9ae377791b904e4e9bbf973b2dee9dd21815a36d59c625d1264b88
Oct 30 07:41:00.695776 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:41:00.695737    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e71af10_fc20_4c4e_8458_93b5f3afd68e.slice/crio-5b89ea7049696937c532f9afd3eb519ba77e463f53cb59fd4620df5ca10d3766 WatchSource:0}: Error finding container 5b89ea7049696937c532f9afd3eb519ba77e463f53cb59fd4620df5ca10d3766: Status 404 returned error can't find the container with id 5b89ea7049696937c532f9afd3eb519ba77e463f53cb59fd4620df5ca10d3766
Oct 30 07:41:00.697934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.697887    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:41:00.699140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.699100    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:41:00.699215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.699178    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:41:00.699215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.699210    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:41:00.733054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:00.733022    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:01.105577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.105344    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" event={"ID":"9e71af10-fc20-4c4e-8458-93b5f3afd68e","Type":"ContainerStarted","Data":"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"}
Oct 30 07:41:01.105577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.105444    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" event={"ID":"9e71af10-fc20-4c4e-8458-93b5f3afd68e","Type":"ContainerStarted","Data":"5b89ea7049696937c532f9afd3eb519ba77e463f53cb59fd4620df5ca10d3766"}
Oct 30 07:41:01.107576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.107543    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" event={"ID":"94d81899-c2b9-4536-9a5b-0202cc6dff08","Type":"ContainerStarted","Data":"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"}
Oct 30 07:41:01.107815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.107788    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:41:01.107932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.107824    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" event={"ID":"94d81899-c2b9-4536-9a5b-0202cc6dff08","Type":"ContainerStarted","Data":"008375da9d9ae377791b904e4e9bbf973b2dee9dd21815a36d59c625d1264b88"}
Oct 30 07:41:01.157406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.157334    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podStartSLOduration=2.157315278 podStartE2EDuration="2.157315278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:01.154604701 +0000 UTC m=+2621.587368607" watchObservedRunningTime="2026-10-30 07:41:01.157315278 +0000 UTC m=+2621.590079172"
Oct 30 07:41:01.229919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.229362    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" podStartSLOduration=2.229339815 podStartE2EDuration="2.229339815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:01.212661184 +0000 UTC m=+2621.645425118" watchObservedRunningTime="2026-10-30 07:41:01.229339815 +0000 UTC m=+2621.662103709"
Oct 30 07:41:01.376021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.375984    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:01.378201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.378160    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:01.378422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:01.378254    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:02.379320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:02.375450    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:02.379320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:02.375565    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:02.697078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:02.696992    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:03.388320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:03.387930    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:03.388320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:03.387993    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:04.394318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:04.388797    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:04.394318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:04.388867    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:04.666786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:04.666680    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:04.993868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:04.993784    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:41:05.397319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:05.396943    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:05.397319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:05.397009    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:06.382321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:06.376752    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:06.382321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:06.376818    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:06.693030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:06.692937    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:07.376240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:07.376203    6738 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused" start-of-body=
Oct 30 07:41:07.376609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:07.376267    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.105:15021/healthz/ready\": dial tcp 10.133.1.105:15021: connect: connection refused"
Oct 30 07:41:07.521194 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:41:07.521149    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:41:08.406470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:08.406393    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:08.407984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:08.407954    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:08.408690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:08.408662    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:41:08.732173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:08.732078    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:10.648856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:10.648820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:12.119841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:12.119809    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:41:12.569408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:12.569367    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:14.329726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:14.329688    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:16.259922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:16.259886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:18.117767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:18.117726    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:19.980510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:19.980475    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:21.948447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:21.948376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:23.035206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:23.035166    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"]
Oct 30 07:41:23.046558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:23.046522    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77387f9c-mpvr2"]
Oct 30 07:41:23.692781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:23.692730    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6937f39-19a5-4ec8-9b83-358534fd1a79" path="/var/lib/kubelet/pods/a6937f39-19a5-4ec8-9b83-358534fd1a79/volumes"
Oct 30 07:41:23.734192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:23.734158    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:25.633604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:25.633558    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:27.453434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:27.453394    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:29.336466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:29.336427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:31.236083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:31.236053    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:33.005217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:33.005170    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:33.117888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:33.117845    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:35.040719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:35.040682    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:37.044365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:37.044205    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:37.566954 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:41:37.566907    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:41:39.327881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.327836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:39.774026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.773989    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"]
Oct 30 07:41:39.780422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.780393    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.797960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.797923    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"]
Oct 30 07:41:39.802521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.802488    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:41:39.802521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.802504    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:41:39.886155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.886117    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.886373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.886331    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.886498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.886482    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwkt\" (UniqueName: \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.994555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.994513    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.994788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.994724    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.994926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.994908    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwkt\" (UniqueName: \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.995459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.995434    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:39.997422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:39.997395    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:40.019370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:40.019339    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwkt\" (UniqueName: \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\") pod \"maas-api-revoke-sub-3ba310e5-x9tfw\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:40.100591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:40.100505    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:40.409197 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:41:40.409167    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09392c7f_169f_4922_9e68_aa5284dd4963.slice/crio-8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01 WatchSource:0}: Error finding container 8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01: Status 404 returned error can't find the container with id 8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01
Oct 30 07:41:40.410340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:40.410313    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"]
Oct 30 07:41:40.484756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:40.484717    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw" event={"ID":"09392c7f-169f-4922-9e68-aa5284dd4963","Type":"ContainerStarted","Data":"8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01"}
Oct 30 07:41:41.250617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:41.250586    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:41.495003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:41.494974    6738 generic.go:356] "Generic (PLEG): container finished" podID="09392c7f-169f-4922-9e68-aa5284dd4963" containerID="02a2f2aa1c1b9e02b8db10d6c4d9efa42c98645f67120d441008bbb386635ea6" exitCode=0
Oct 30 07:41:41.495481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:41.495059    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw" event={"ID":"09392c7f-169f-4922-9e68-aa5284dd4963","Type":"ContainerDied","Data":"02a2f2aa1c1b9e02b8db10d6c4d9efa42c98645f67120d441008bbb386635ea6"}
Oct 30 07:41:42.669760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.669731    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:42.748217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.748180    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\") pod \"09392c7f-169f-4922-9e68-aa5284dd4963\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") "
Oct 30 07:41:42.748471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.748447    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\" (UniqueName: \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\") pod \"09392c7f-169f-4922-9e68-aa5284dd4963\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") "
Oct 30 07:41:42.748620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.748604    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\") pod \"09392c7f-169f-4922-9e68-aa5284dd4963\" (UID: \"09392c7f-169f-4922-9e68-aa5284dd4963\") "
Oct 30 07:41:42.754177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.749037    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca" pod "09392c7f-169f-4922-9e68-aa5284dd4963" (UID: "09392c7f-169f-4922-9e68-aa5284dd4963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:42.754177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.749550    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:41:42.754177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.750789    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt" pod "09392c7f-169f-4922-9e68-aa5284dd4963" (UID: "09392c7f-169f-4922-9e68-aa5284dd4963"). InnerVolumeSpecName "kube-api-access-bhwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:42.754177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.754121    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca" pod "09392c7f-169f-4922-9e68-aa5284dd4963" (UID: "09392c7f-169f-4922-9e68-aa5284dd4963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:41:42.851583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.851546    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09392c7f-169f-4922-9e68-aa5284dd4963-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:41:42.851583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:42.851584    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwkt\" (UniqueName: \"kubernetes.io/projected/09392c7f-169f-4922-9e68-aa5284dd4963-kube-api-access-bhwkt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:41:43.209030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:43.208996    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:43.515910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:43.515823    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"
Oct 30 07:41:43.515910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:43.515865    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw" event={"ID":"09392c7f-169f-4922-9e68-aa5284dd4963","Type":"ContainerDied","Data":"8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01"}
Oct 30 07:41:43.516128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:43.515921    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf8a9c84273da2e3a626fffd65ac6bfb41619f4dc777ff5bacaedee67df1e01"
Oct 30 07:41:45.060894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:45.060852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:46.836777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:46.836731    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:48.669470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:48.669437    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:48.786959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:48.786919    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:48.917632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:48.917599    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:50.748107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:50.748074    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:51.038459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.038375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:51.171042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.171000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:51.680423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.680389    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:41:51.681323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.681280    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09392c7f-169f-4922-9e68-aa5284dd4963" containerName="revoke-keys"
Oct 30 07:41:51.681400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.681325    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09392c7f-169f-4922-9e68-aa5284dd4963" containerName="revoke-keys"
Oct 30 07:41:51.681455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.681446    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09392c7f-169f-4922-9e68-aa5284dd4963" containerName="revoke-keys"
Oct 30 07:41:51.685687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.685662    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.727320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.721949    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:41:51.804044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.803999    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.804706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.804320    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.804706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.804362    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plgl\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.804706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.804395    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.906819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.906771    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.906819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.906828    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plgl\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.907147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.906861    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.907147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.906993    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.909709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.909678    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.910136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.910115    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.910189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.910137    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:51.935860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:51.935787    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plgl\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\") pod \"payload-processing-external-model-d9775f665-g8x4q\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") " pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:52.029021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.028987    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:52.347427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.347392    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:41:52.347903 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:41:52.347880    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d95f77_8293_436b_aacc_f34aa5400169.slice/crio-bd7d83e9fba8134c59342d047d6b054aefd083c03ad6faa30862f6909c8fcd5e WatchSource:0}: Error finding container bd7d83e9fba8134c59342d047d6b054aefd083c03ad6faa30862f6909c8fcd5e: Status 404 returned error can't find the container with id bd7d83e9fba8134c59342d047d6b054aefd083c03ad6faa30862f6909c8fcd5e
Oct 30 07:41:52.595141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.595040    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" event={"ID":"c6d95f77-8293-436b-aacc-f34aa5400169","Type":"ContainerStarted","Data":"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"}
Oct 30 07:41:52.595141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.595105    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" event={"ID":"c6d95f77-8293-436b-aacc-f34aa5400169","Type":"ContainerStarted","Data":"bd7d83e9fba8134c59342d047d6b054aefd083c03ad6faa30862f6909c8fcd5e"}
Oct 30 07:41:52.595141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.595134    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:41:52.638977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.638921    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podStartSLOduration=1.6389048370000001 podStartE2EDuration="1.638904837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:52.636884744 +0000 UTC m=+2673.069648636" watchObservedRunningTime="2026-10-30 07:41:52.638904837 +0000 UTC m=+2673.071668729"
Oct 30 07:41:52.739584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.739546    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:52.921320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:52.920844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:53.059319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:53.056922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:53.354387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:53.354262    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:53.674492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:53.674457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:53.794009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:53.793978    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:54.846724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:54.846681    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:56.687645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:56.687603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:58.554821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:58.554783    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:58.845532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:58.845444    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:59.827074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:59.827025    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:41:59.948761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:41:59.948719    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:00.867227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:00.867199    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:01.586174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.586133    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:01.837884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.837788    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:42:01.843334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.843306    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:01.864898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.864868    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:42:01.871457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871430    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"routing-overlay\""
Oct 30 07:42:01.871881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871510    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"openshift-service-ca.crt\""
Oct 30 07:42:01.871881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871436    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-977360dd41b9\""
Oct 30 07:42:01.871881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871508    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-e2e--e56a4edc-dockercfg-gg9nv\""
Oct 30 07:42:01.871881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871601    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:42:01.871881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.871608    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"kube-root-ca.crt\""
Oct 30 07:42:01.933799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.933759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:01.933799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.933799    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:01.934104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.933961    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:01.934104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:01.934010    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ddw\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.048468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.048421    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.051180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.051111    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.051917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.051893    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.052122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.052105    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ddw\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.052375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.049944    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.053062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.053013    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.055869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.055844    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.100392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.097768    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ddw\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.166902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.166868    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.488831 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:02.488794    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7d69f_11f1_4f9b_aa29_eeda025cadf9.slice/crio-ec64155ad4d8be2fb639b5abce7a662221d2ac1ac9803366917976a404f923f8 WatchSource:0}: Error finding container ec64155ad4d8be2fb639b5abce7a662221d2ac1ac9803366917976a404f923f8: Status 404 returned error can't find the container with id ec64155ad4d8be2fb639b5abce7a662221d2ac1ac9803366917976a404f923f8
Oct 30 07:42:02.490175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.490149    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:42:02.620638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.620605    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:02.866391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.865888    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" event={"ID":"aff7d69f-11f1-4f9b-aa29-eeda025cadf9","Type":"ContainerStarted","Data":"24a9e1715361cccfa28f0fccf2e19f2209a4cced9f6b47e1af4e4206402771c1"}
Oct 30 07:42:02.866391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.865972    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" event={"ID":"aff7d69f-11f1-4f9b-aa29-eeda025cadf9","Type":"ContainerStarted","Data":"ec64155ad4d8be2fb639b5abce7a662221d2ac1ac9803366917976a404f923f8"}
Oct 30 07:42:02.866391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.866026    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:02.880707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.880676    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:42:02.885669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.885646    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:02.901695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.901666    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8f1df\""
Oct 30 07:42:02.901881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.901693    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"agc-cred-b82c8e26906c\""
Oct 30 07:42:02.901881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.901757    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"openshift-service-ca.crt\""
Oct 30 07:42:02.901881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.901806    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"kube-root-ca.crt\""
Oct 30 07:42:02.902470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.902445    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"routing-overlay\""
Oct 30 07:42:02.902916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.901880    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8f1df\"/\"payload-processing-external-model-e2e--97d1e837-dockercfg-29xk2\""
Oct 30 07:42:02.903728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.903707    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:42:02.965481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.965420    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" podStartSLOduration=1.965401642 podStartE2EDuration="1.965401642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:02.923541719 +0000 UTC m=+2683.356305612" watchObservedRunningTime="2026-10-30 07:42:02.965401642 +0000 UTC m=+2683.398165538"
Oct 30 07:42:02.977753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.977716    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qdq\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:02.977949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.977871    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:02.978050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.978018    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:02.978100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.978064    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:02.982165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:02.982138    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:03.094338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.090837    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qdq\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.097325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.095024    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.097325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.095306    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.097325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.095354    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.118373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.114538    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.118373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.115174    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.139327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.122178    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.158315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.146030    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qdq\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\") pod \"payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") " pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.207315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.206920    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.585450 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:03.585418    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72500675_1c7e_4ca0_9c35_22428f8eb1b9.slice/crio-5c2dbc29feb0c6fdd6e2a9a71e0dd17dbef9bf8c4b8dd49adb3a9e4316fc67d8 WatchSource:0}: Error finding container 5c2dbc29feb0c6fdd6e2a9a71e0dd17dbef9bf8c4b8dd49adb3a9e4316fc67d8: Status 404 returned error can't find the container with id 5c2dbc29feb0c6fdd6e2a9a71e0dd17dbef9bf8c4b8dd49adb3a9e4316fc67d8
Oct 30 07:42:03.587361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.587339    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:42:03.648382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.648284    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:42:03.876600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.876560    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" event={"ID":"72500675-1c7e-4ca0-9c35-22428f8eb1b9","Type":"ContainerStarted","Data":"28ca196b114eead683e019d31158e365a727d4303aacad5faf4cff4c744a9dd4"}
Oct 30 07:42:03.876821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.876638    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" event={"ID":"72500675-1c7e-4ca0-9c35-22428f8eb1b9","Type":"ContainerStarted","Data":"5c2dbc29feb0c6fdd6e2a9a71e0dd17dbef9bf8c4b8dd49adb3a9e4316fc67d8"}
Oct 30 07:42:03.876899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.876875    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:03.921684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:03.921634    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" podStartSLOduration=1.921620306 podStartE2EDuration="1.921620306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:03.918891313 +0000 UTC m=+2684.351655202" watchObservedRunningTime="2026-10-30 07:42:03.921620306 +0000 UTC m=+2684.354384200"
Oct 30 07:42:04.864088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:04.864050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:06.631380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:06.631341    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:07.536122 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:07.536078    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:42:08.568038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:08.567995    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:09.802121 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:09.802078    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:09.802121 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:09.802110    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:09.802801 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:09.802239    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:10.302215284 +0000 UTC m=+2690.734979157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:10.308537 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:10.308505    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:10.308537 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:10.308534    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:10.308885 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:10.308605    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:11.30858231 +0000 UTC m=+2691.741346208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:10.442187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:10.442140    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:11.327691 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:11.327646    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:11.327691 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:11.327680    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:11.328249 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:11.327763    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:13.32774284 +0000 UTC m=+2693.760506710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:12.321568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:12.321521    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:13.357031 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:13.356994    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:13.357031 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:13.357028    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:13.357535 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:13.357099    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:17.357083232 +0000 UTC m=+2697.789847119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:13.645977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:13.645937    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:42:13.880117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:13.880084    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:42:14.129774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:14.129739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:14.887965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:14.887933    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:42:15.963274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:15.963234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:17.450496 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:17.450465    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:17.450496 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:17.450491    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:17.450981 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:17.450556    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:25.450540423 +0000 UTC m=+2705.883304307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:17.818282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:17.818173    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:19.657070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:19.657028    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:21.949166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:21.949107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:22.050102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:22.050066    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"]
Oct 30 07:42:22.103317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:22.088457    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1640-g8pmf"]
Oct 30 07:42:22.623320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:22.623093    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:23.647117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:23.647053    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:42:23.718709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:23.718661    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d71610-9154-4b6a-8cd5-4e3e644c095b" path="/var/lib/kubelet/pods/64d71610-9154-4b6a-8cd5-4e3e644c095b/volumes"
Oct 30 07:42:24.265993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:24.265954    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:24.805332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:24.804413    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7f465 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:42:24.805332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:24.804503    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7f465" podUID="384cf69e-4f91-4b58-8001-875083896c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:42:25.539910 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:25.539873    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:25.539910 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:25.539901    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:25.540169 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:25.539955    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:42:41.539938816 +0000 UTC m=+2721.972702707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:26.355702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:26.355666    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:27.808746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.808703    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"]
Oct 30 07:42:27.814940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.814913    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.831079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.831043    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"]
Oct 30 07:42:27.857794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.837712    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:42:27.857794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.837846    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:42:27.877867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.877812    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.878078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.878035    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.878160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.878108    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.980086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.980051    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.980357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.980141    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.980357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.980190    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.980890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.980861    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:27.982740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:27.982720    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:28.004101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:28.004073    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-bc7dfd28-nxnsb\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:28.137128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:28.137095    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:28.234937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:28.234896    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:28.449109 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:28.449075    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97afbe8_0256_4559_89b3_7e291813240c.slice/crio-ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa WatchSource:0}: Error finding container ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa: Status 404 returned error can't find the container with id ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa
Oct 30 07:42:28.450830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:28.450802    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"]
Oct 30 07:42:29.356482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:29.356445    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb" event={"ID":"a97afbe8-0256-4559-89b3-7e291813240c","Type":"ContainerStarted","Data":"8c05f6efb324eeb6cee4f4c4934334140f58a1b996f9ae68c7bca8319bc780b5"}
Oct 30 07:42:29.356935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:29.356509    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb" event={"ID":"a97afbe8-0256-4559-89b3-7e291813240c","Type":"ContainerStarted","Data":"ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa"}
Oct 30 07:42:29.399408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:29.399352    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb" podStartSLOduration=2.399334802 podStartE2EDuration="2.399334802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:29.397569647 +0000 UTC m=+2709.830333539" watchObservedRunningTime="2026-10-30 07:42:29.399334802 +0000 UTC m=+2709.832098695"
Oct 30 07:42:29.468938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:29.468903    6738 scope.go:122] "RemoveContainer" containerID="fa7e3219133ab53c40133b87791ef88b524b7bd4bd850dd3db31efa99f56ca2c"
Oct 30 07:42:30.076305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:30.076252    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:30.367793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:30.367707    6738 generic.go:356] "Generic (PLEG): container finished" podID="a97afbe8-0256-4559-89b3-7e291813240c" containerID="8c05f6efb324eeb6cee4f4c4934334140f58a1b996f9ae68c7bca8319bc780b5" exitCode=0
Oct 30 07:42:30.367793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:30.367741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb" event={"ID":"a97afbe8-0256-4559-89b3-7e291813240c","Type":"ContainerDied","Data":"8c05f6efb324eeb6cee4f4c4934334140f58a1b996f9ae68c7bca8319bc780b5"}
Oct 30 07:42:31.563384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.563354    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:31.657979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.657865    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\") pod \"a97afbe8-0256-4559-89b3-7e291813240c\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") "
Oct 30 07:42:31.658180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.658088    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\") pod \"a97afbe8-0256-4559-89b3-7e291813240c\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") "
Oct 30 07:42:31.658180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.658171    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\") pod \"a97afbe8-0256-4559-89b3-7e291813240c\" (UID: \"a97afbe8-0256-4559-89b3-7e291813240c\") "
Oct 30 07:42:31.658581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.658549    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca" pod "a97afbe8-0256-4559-89b3-7e291813240c" (UID: "a97afbe8-0256-4559-89b3-7e291813240c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:42:31.659131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.659106    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:31.660435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.660391    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca" pod "a97afbe8-0256-4559-89b3-7e291813240c" (UID: "a97afbe8-0256-4559-89b3-7e291813240c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:42:31.660891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.660863    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j" pod "a97afbe8-0256-4559-89b3-7e291813240c" (UID: "a97afbe8-0256-4559-89b3-7e291813240c"). InnerVolumeSpecName "kube-api-access-scz5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:42:31.760707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.760666    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97afbe8-0256-4559-89b3-7e291813240c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:31.760707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.760698    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/a97afbe8-0256-4559-89b3-7e291813240c-kube-api-access-scz5j\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:31.924574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:31.924467    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:32.386726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:32.386699    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"
Oct 30 07:42:32.386915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:32.386740    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb" event={"ID":"a97afbe8-0256-4559-89b3-7e291813240c","Type":"ContainerDied","Data":"ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa"}
Oct 30 07:42:32.386915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:32.386788    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef0a46a3b55df793d16b39b7c7887e81c33d076f65ecd41df14e8f56eaba27fa"
Oct 30 07:42:33.130330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:33.130279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:33.252125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:33.252091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:33.646573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:33.646537    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:42:33.847960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:33.847925    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:35.759282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:35.759248    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:37.524504 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:37.524463    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:42:37.626921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:37.626882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:39.106407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.106374    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:39.231688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.229722    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:39.627862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.627824    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:39.937515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.937421    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:42:39.950375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.950341    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:42:39.951757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:39.951724    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:42:40.029264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:40.029218    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:41.265513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:41.265479    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:41.608449 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:41.608359    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-031aedf881ae: secret "agc-cred-031aedf881ae" not found
Oct 30 07:42:41.608449 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:41.608390    6738 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c204db61a2ae: secret "agc-cred-c204db61a2ae" not found
Oct 30 07:42:41.608449 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:41.608400    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj: [secret "agc-cred-031aedf881ae" not found, secret "agc-cred-c204db61a2ae" not found]
Oct 30 07:42:41.608796 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:41.608456    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials podName:09c7b92d-9336-4f09-ad90-cab59eaa55b2 nodeName:}" failed. No retries permitted until 2026-10-30 07:43:13.608441999 +0000 UTC m=+2754.041205885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials") pod "payload-processing-external-model-6cc568d7d6-4dcvj" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2") : [secret "agc-cred-031aedf881ae" not found, secret "agc-cred-c204db61a2ae" not found]
Oct 30 07:42:41.634206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:41.634175    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:41.786616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:41.786578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:41.988015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:41.987959    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:43.474051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:43.473996    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:44.565112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:44.565071    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:44.938412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:44.938347    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerName="payload-processing" containerID="cri-o://6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8" gracePeriod=25
Oct 30 07:42:44.950927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:44.950869    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing" containerID="cri-o://1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365" gracePeriod=25
Oct 30 07:42:44.973421 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:44.973394    6738 logging.go:55] [core] [Channel #10258 SubChannel #10259]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.103:50052", ServerName: "10.133.1.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.103:50052: connect: connection refused"
Oct 30 07:42:45.154668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.154536    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:45.259620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.259593    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:42:45.287518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.287488    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\") pod \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") "
Oct 30 07:42:45.287792 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.287771    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\") pod \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") "
Oct 30 07:42:45.287891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.287859    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\") pod \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") "
Oct 30 07:42:45.287958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.287897    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\") pod \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\" (UID: \"09c7b92d-9336-4f09-ad90-cab59eaa55b2\") "
Oct 30 07:42:45.288013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.287950    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume" pod "09c7b92d-9336-4f09-ad90-cab59eaa55b2" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:42:45.288242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.288217    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume" pod "09c7b92d-9336-4f09-ad90-cab59eaa55b2" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:42:45.288555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.288522    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.288678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.288558    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09c7b92d-9336-4f09-ad90-cab59eaa55b2-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.290366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.290273    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t" pod "09c7b92d-9336-4f09-ad90-cab59eaa55b2" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2"). InnerVolumeSpecName "kube-api-access-g6v6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:42:45.290453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.290375    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials" pod "09c7b92d-9336-4f09-ad90-cab59eaa55b2" (UID: "09c7b92d-9336-4f09-ad90-cab59eaa55b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:42:45.291509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.291493    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:42:45.389144 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389102    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\") pod \"c6d95f77-8293-436b-aacc-f34aa5400169\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") "
Oct 30 07:42:45.389372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389178    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\") pod \"c6d95f77-8293-436b-aacc-f34aa5400169\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") "
Oct 30 07:42:45.389372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389242    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\") pod \"c6d95f77-8293-436b-aacc-f34aa5400169\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") "
Oct 30 07:42:45.389488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389455    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\") pod \"c6d95f77-8293-436b-aacc-f34aa5400169\" (UID: \"c6d95f77-8293-436b-aacc-f34aa5400169\") "
Oct 30 07:42:45.389608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389577    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume" pod "c6d95f77-8293-436b-aacc-f34aa5400169" (UID: "c6d95f77-8293-436b-aacc-f34aa5400169"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:42:45.389971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.389941    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume" pod "c6d95f77-8293-436b-aacc-f34aa5400169" (UID: "c6d95f77-8293-436b-aacc-f34aa5400169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:42:45.390410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.390382    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.390410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.390409    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6v6t\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-kube-api-access-g6v6t\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.390554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.390424    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09c7b92d-9336-4f09-ad90-cab59eaa55b2-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.390554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.390437    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6d95f77-8293-436b-aacc-f34aa5400169-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.391743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.391720    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials" pod "c6d95f77-8293-436b-aacc-f34aa5400169" (UID: "c6d95f77-8293-436b-aacc-f34aa5400169"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:42:45.392085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.392070    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl" pod "c6d95f77-8293-436b-aacc-f34aa5400169" (UID: "c6d95f77-8293-436b-aacc-f34aa5400169"). InnerVolumeSpecName "kube-api-access-4plgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:42:45.492395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.492290    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plgl\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-kube-api-access-4plgl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.492395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.492340    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6d95f77-8293-436b-aacc-f34aa5400169-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:42:45.524039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524001    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:42:45.524930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524910    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing"
Oct 30 07:42:45.524930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524930    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing"
Oct 30 07:42:45.525111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524962    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a97afbe8-0256-4559-89b3-7e291813240c" containerName="revoke-keys"
Oct 30 07:42:45.525111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524968    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97afbe8-0256-4559-89b3-7e291813240c" containerName="revoke-keys"
Oct 30 07:42:45.525111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524981    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerName="payload-processing"
Oct 30 07:42:45.525111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.524987    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerName="payload-processing"
Oct 30 07:42:45.525372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.525139    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" containerName="payload-processing"
Oct 30 07:42:45.525372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.525154    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97afbe8-0256-4559-89b3-7e291813240c" containerName="revoke-keys"
Oct 30 07:42:45.525372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.525161    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerName="payload-processing"
Oct 30 07:42:45.529986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.529957    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.562076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.562024    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:42:45.582402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.582368    6738 generic.go:356] "Generic (PLEG): container finished" podID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerID="6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8" exitCode=0
Oct 30 07:42:45.583064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.582626    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"
Oct 30 07:42:45.587686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.587656    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4ab37e-openshift-default-dockercfg-4l69h\""
Oct 30 07:42:45.588189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.582673    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" event={"ID":"09c7b92d-9336-4f09-ad90-cab59eaa55b2","Type":"ContainerDied","Data":"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"}
Oct 30 07:42:45.588289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.588248    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" event={"ID":"09c7b92d-9336-4f09-ad90-cab59eaa55b2","Type":"ContainerDied","Data":"f7c8c763b9ba37f23c8206f423adc49998d5d6dda9e59fc7d3aab844c61bb998"}
Oct 30 07:42:45.588411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.588322    6738 scope.go:122] "RemoveContainer" containerID="6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"
Oct 30 07:42:45.591739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.591711    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.592898    6738 generic.go:356] "Generic (PLEG): container finished" podID="c6d95f77-8293-436b-aacc-f34aa5400169" containerID="1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365" exitCode=0
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.593143    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" event={"ID":"c6d95f77-8293-436b-aacc-f34aa5400169","Type":"ContainerDied","Data":"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"}
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.593195    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q" event={"ID":"c6d95f77-8293-436b-aacc-f34aa5400169","Type":"ContainerDied","Data":"bd7d83e9fba8134c59342d047d6b054aefd083c03ad6faa30862f6909c8fcd5e"}
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.593277    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595018    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595132    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595197    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595255    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595356    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595389    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595456    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscx9\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595535    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.598226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.595614    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.616373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.616342    6738 scope.go:122] "RemoveContainer" containerID="6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"
Oct 30 07:42:45.616785 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:45.616756    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8\": container with ID starting with 6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8 not found: ID does not exist" containerID="6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"
Oct 30 07:42:45.616893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.616799    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8"} err="failed to get container status \"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8\": rpc error: code = NotFound desc = could not find container \"6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8\": container with ID starting with 6fe6ded27de886650d6bd805f9101fa601e355be768195c55f928729dc00d6b8 not found: ID does not exist"
Oct 30 07:42:45.616893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.616854    6738 scope.go:122] "RemoveContainer" containerID="1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"
Oct 30 07:42:45.629771 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.629743    6738 scope.go:122] "RemoveContainer" containerID="1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"
Oct 30 07:42:45.630130 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:42:45.630110    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365\": container with ID starting with 1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365 not found: ID does not exist" containerID="1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"
Oct 30 07:42:45.630214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.630141    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365"} err="failed to get container status \"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365\": rpc error: code = NotFound desc = could not find container \"1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365\": container with ID starting with 1acc47d436ef4471a2d8b77ab40150bb8a5f640281ffee8b1b2abb654e499365 not found: ID does not exist"
Oct 30 07:42:45.651159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.651128    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:42:45.656020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.655998    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d9775f665-g8x4q"]
Oct 30 07:42:45.693568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.693522    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d95f77-8293-436b-aacc-f34aa5400169" path="/var/lib/kubelet/pods/c6d95f77-8293-436b-aacc-f34aa5400169/volumes"
Oct 30 07:42:45.693947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.693928    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:42:45.694690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.694666    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj"]
Oct 30 07:42:45.701456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.701418    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.701683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.701663    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.701906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.701887    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscx9\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.702080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.702064    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.702244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.702229    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.702438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.702422    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.702767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.702745    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.702947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.702930    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.703097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.703082    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.703794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.703769    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.738319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.729315    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.738319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.729635    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.738319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.735878    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.761384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.747081    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.786320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.767657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.786320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.770576    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.811330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.806958    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.857320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.814904    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscx9\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.857320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.817655    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:45.880317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.869619    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:45.977712 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:45.977680    6738 logging.go:55] [core] [Channel #10258 SubChannel #10259]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.103:50052", ServerName: "10.133.1.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.103:50052: operation was canceled"
Oct 30 07:42:45.977945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.977564    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6cc568d7d6-4dcvj" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.103:50052\" within 1s: context deadline exceeded"
Oct 30 07:42:45.983684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:45.983588    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:46.289542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.289506    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:46.357125 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:46.357097    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda733a20c_bf1c_460b_86a6_2a849f7c17d8.slice/crio-59a936243ef46e89a887ec4b8f5b13439e1178518dab0a2218de1d3acc2d828d WatchSource:0}: Error finding container 59a936243ef46e89a887ec4b8f5b13439e1178518dab0a2218de1d3acc2d828d: Status 404 returned error can't find the container with id 59a936243ef46e89a887ec4b8f5b13439e1178518dab0a2218de1d3acc2d828d
Oct 30 07:42:46.357593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.357563    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:42:46.360218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.360180    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:42:46.360332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.360252    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:42:46.360332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.360282    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:42:46.415375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.415337    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:46.606961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.606856    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerStarted","Data":"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"}
Oct 30 07:42:46.606961 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.606913    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerStarted","Data":"59a936243ef46e89a887ec4b8f5b13439e1178518dab0a2218de1d3acc2d828d"}
Oct 30 07:42:46.653197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.653102    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podStartSLOduration=1.653084075 podStartE2EDuration="1.653084075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:46.653064167 +0000 UTC m=+2727.085828065" watchObservedRunningTime="2026-10-30 07:42:46.653084075 +0000 UTC m=+2727.085847971"
Oct 30 07:42:46.871104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:46.871010    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:47.692924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:47.692889    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c7b92d-9336-4f09-ad90-cab59eaa55b2" path="/var/lib/kubelet/pods/09c7b92d-9336-4f09-ad90-cab59eaa55b2/volumes"
Oct 30 07:42:47.877449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:47.877414    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:48.166662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:48.166625    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:48.626540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:48.626514    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:48.627468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:48.627446    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:42:49.082664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:49.082578    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:49.196677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:49.196643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:49.522798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:49.512447    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:49.654352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:49.651091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:50.260784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:50.260745    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:51.303752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.303720    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:51.418930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.418891    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:51.470252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.470218    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:42:51.475644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.475617    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.493403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.493375    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4ab37e-dockercfg-kld95\""
Oct 30 07:42:51.493692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.493666    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4ab37e\""
Oct 30 07:42:51.497622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.497595    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:42:51.571387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.563400    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.571387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.563751    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.711793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.711747    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.712443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.712409    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.725391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.725350    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.780282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.780241    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\") pod \"payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:51.797624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:51.797577    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:52.130674 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:42:52.130634    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3882ab1d_f3f6_4344_95bf_45474a27e29a.slice/crio-f53a97fb9e49d608a228f2f2376143afbbe10eaf2ad25b6014958cb8f72233ed WatchSource:0}: Error finding container f53a97fb9e49d608a228f2f2376143afbbe10eaf2ad25b6014958cb8f72233ed: Status 404 returned error can't find the container with id f53a97fb9e49d608a228f2f2376143afbbe10eaf2ad25b6014958cb8f72233ed
Oct 30 07:42:52.131480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.131455    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:42:52.373366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.373327    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:52.795106 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.795068    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" event={"ID":"3882ab1d-f3f6-4344-95bf-45474a27e29a","Type":"ContainerStarted","Data":"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"}
Oct 30 07:42:52.795322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.795125    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" event={"ID":"3882ab1d-f3f6-4344-95bf-45474a27e29a","Type":"ContainerStarted","Data":"f53a97fb9e49d608a228f2f2376143afbbe10eaf2ad25b6014958cb8f72233ed"}
Oct 30 07:42:52.795322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.795155    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:42:52.841582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:52.841523    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" podStartSLOduration=1.8415088979999998 podStartE2EDuration="1.841508898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:52.837122556 +0000 UTC m=+2733.269886449" watchObservedRunningTime="2026-10-30 07:42:52.841508898 +0000 UTC m=+2733.274272791"
Oct 30 07:42:54.259058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:54.259021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:56.101423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:56.101385    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:57.930829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:57.930782    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:42:59.932208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:42:59.932173    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:01.918899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:01.918865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:03.842515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:03.842477    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:43:04.190319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:04.183501    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:06.257545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:06.257508    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:07.521349 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:43:07.521286    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:43:08.156885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:08.156846    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:10.019950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:10.019917    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:11.881850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:11.881809    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:13.033401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:13.033366    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"]
Oct 30 07:43:13.045487 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:13.045448    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504f37b-2kv4h"]
Oct 30 07:43:13.693088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:13.693053    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51" path="/var/lib/kubelet/pods/9cd04e5d-78a3-4b87-bfe3-fdbb7b963c51/volumes"
Oct 30 07:43:13.760509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:13.760472    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:15.652457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:15.652401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:17.508787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:17.508747    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:19.359873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:19.359841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:20.031492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:20.031445    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"]
Oct 30 07:43:20.043722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:20.043687    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51ff70bc-tpsnz"]
Oct 30 07:43:21.152239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:21.152201    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:21.692672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:21.692635    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115f63c2-4dcd-4e32-bbc8-0e2d7a00360f" path="/var/lib/kubelet/pods/115f63c2-4dcd-4e32-bbc8-0e2d7a00360f/volumes"
Oct 30 07:43:23.011309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:23.011266    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:24.037576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:24.037535    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"]
Oct 30 07:43:24.049656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:24.049623    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e0cfca-cbstl"]
Oct 30 07:43:24.792897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:24.792841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:25.033238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:25.033198    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"]
Oct 30 07:43:25.045324 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:25.045238    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86289c6-dzlcg"]
Oct 30 07:43:25.692550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:25.692512    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3ebb3f-6204-49e9-95d5-88a57e5aebc6" path="/var/lib/kubelet/pods/1d3ebb3f-6204-49e9-95d5-88a57e5aebc6/volumes"
Oct 30 07:43:25.693027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:25.693012    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539d15bd-4568-4cf0-98e6-45af7a63376e" path="/var/lib/kubelet/pods/539d15bd-4568-4cf0-98e6-45af7a63376e/volumes"
Oct 30 07:43:26.656086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:26.656037    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:28.642476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:28.642437    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:29.505529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:29.505485    6738 scope.go:122] "RemoveContainer" containerID="969f4afe3e73badc58fd89ff45774a35cd325386cea407c8e69ac17f14b84c06"
Oct 30 07:43:30.427001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:30.426950    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:32.265159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:32.265112    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:33.256360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:33.256319    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:34.249029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:34.248981    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:36.033566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:36.033529    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"]
Oct 30 07:43:36.044865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:36.044830    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107fb25d-fg6kd"]
Oct 30 07:43:36.216829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:36.216791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:37.520055 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:43:37.520014    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:43:37.692526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:37.692491    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786eb420-ea6c-4025-b8e1-09d36e331662" path="/var/lib/kubelet/pods/786eb420-ea6c-4025-b8e1-09d36e331662/volumes"
Oct 30 07:43:38.021724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:38.021684    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:38.132462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:38.132427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:38.250901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:38.250858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:39.033015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:39.032978    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"]
Oct 30 07:43:39.045339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:39.045305    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf64ddc-w77qr"]
Oct 30 07:43:39.732078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:39.732042    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4482d977-9965-4541-9ccd-2535df24055b" path="/var/lib/kubelet/pods/4482d977-9965-4541-9ccd-2535df24055b/volumes"
Oct 30 07:43:40.146894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:40.146852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:41.994710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:41.994678    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:42.112419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:42.112383    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:44.041036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:44.041005    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:46.028848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:46.028812    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:47.925104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:47.925057    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:49.724472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:49.724431    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:51.494749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:51.494714    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:53.460006 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:53.459968    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:55.233856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:55.233814    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:57.038819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:57.038778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:43:58.953773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:43:58.953733    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:01.152996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:01.152945    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:03.165764 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:03.165723    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:04.201616 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.201577    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"]
Oct 30 07:44:04.207133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.207107    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.223776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.223741    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"]
Oct 30 07:44:04.227759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.227727    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:44:04.228000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.227785    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:44:04.311036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.310999    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.311226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.311204    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.311279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.311267    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wkk\" (UniqueName: \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.412503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.412463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.412722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.412625    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wkk\" (UniqueName: \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.412834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.412818    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.413179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.413159    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.415221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.415196    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.436729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.436699    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wkk\" (UniqueName: \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\") pod \"maas-api-revoke-sub-261f509c-v8bxm\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.527866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.527779    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:04.697760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.697614    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:04.809612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.809526    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:04.963338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:04.941263    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"]
Oct 30 07:44:05.825368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:05.825326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm" event={"ID":"76ef30a5-18db-4bed-95a6-963facbcd2ea","Type":"ContainerStarted","Data":"a87d4205326fa46e73037af7088ac3092e86e5db66db976f7c8c9f558a6711e6"}
Oct 30 07:44:06.294573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.294527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:06.612307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.612263    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:44:06.618578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.618557    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.633392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.633357    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:44:06.641025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.640534    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e27e0c\""
Oct 30 07:44:06.641025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.640554    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"openshift-service-ca.crt\""
Oct 30 07:44:06.641025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.640602    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"agc-cred-4a4b01b21fc4\""
Oct 30 07:44:06.641025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.640643    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"agc-cred-17bac3af1751\""
Oct 30 07:44:06.647328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.647257    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"payload-processing-external-model-e2e--e3e25a9c-dockercfg-bdjwm\""
Oct 30 07:44:06.647498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.647368    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"kube-root-ca.crt\""
Oct 30 07:44:06.647498 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.647432    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"routing-overlay\""
Oct 30 07:44:06.760373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.760334    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.760553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.760510    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbrd\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.760712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.760689    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.761076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.761050    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.836748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.836707    6738 generic.go:356] "Generic (PLEG): container finished" podID="76ef30a5-18db-4bed-95a6-963facbcd2ea" containerID="4c23f1fd9ab5824c73eafbb8ad3f55a1210dbc43dc54cc6bc3902ffa6148b6f9" exitCode=0
Oct 30 07:44:06.837191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.836790    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm" event={"ID":"76ef30a5-18db-4bed-95a6-963facbcd2ea","Type":"ContainerDied","Data":"4c23f1fd9ab5824c73eafbb8ad3f55a1210dbc43dc54cc6bc3902ffa6148b6f9"}
Oct 30 07:44:06.862978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.862890    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.862978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.862951    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.863234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.863062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbrd\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.863234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.863174    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.863880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.863842    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.864059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.864039    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.866154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.866130    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.886917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.886887    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbrd\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:06.975819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:06.975787    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:07.298205 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:07.298159    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad33057_8d5b_46ea_98ea_690880de6954.slice/crio-8344e1b5259a99b8efe1a6f040e5aaf304600b5ed05ede1f011ffacbaf04f1c3 WatchSource:0}: Error finding container 8344e1b5259a99b8efe1a6f040e5aaf304600b5ed05ede1f011ffacbaf04f1c3: Status 404 returned error can't find the container with id 8344e1b5259a99b8efe1a6f040e5aaf304600b5ed05ede1f011ffacbaf04f1c3
Oct 30 07:44:07.298456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.298436    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:44:07.426788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.426752    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:44:07.432595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.432570    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.447604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.447575    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"agc-cred-1f7990b3c7c7\""
Oct 30 07:44:07.447788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.447775    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"routing-overlay\""
Oct 30 07:44:07.451202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.451176    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:44:07.453704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.453571    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8d0c2f\""
Oct 30 07:44:07.453704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.453640    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"payload-processing-external-model-e2e--d43b6e68-dockercfg-pzntb\""
Oct 30 07:44:07.453704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.453667    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"kube-root-ca.crt\""
Oct 30 07:44:07.453928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.453722    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"agc-cred-f5a4d3d71b6b\""
Oct 30 07:44:07.453928 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.453790    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8d0c2f\"/\"openshift-service-ca.crt\""
Oct 30 07:44:07.523115 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:07.523081    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:44:07.576247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.576151    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6mw\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.576416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.576395    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.576493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.576449    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.576493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.576471    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.678471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.678428    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6mw\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.678681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.678627    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.678757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.678688    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.678757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.678720    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.679450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.679425    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.679590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.679571    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.681249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.681227    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.702902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.702861    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6mw\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.751178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.751136    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:07.861097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.861058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" event={"ID":"2ad33057-8d5b-46ea-98ea-690880de6954","Type":"ContainerStarted","Data":"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"}
Oct 30 07:44:07.861586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.861129    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" event={"ID":"2ad33057-8d5b-46ea-98ea-690880de6954","Type":"ContainerStarted","Data":"8344e1b5259a99b8efe1a6f040e5aaf304600b5ed05ede1f011ffacbaf04f1c3"}
Oct 30 07:44:07.861586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.861178    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:07.921724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:07.921652    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" podStartSLOduration=1.921631431 podStartE2EDuration="1.921631431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:07.919545621 +0000 UTC m=+2808.352309523" watchObservedRunningTime="2026-10-30 07:44:07.921631431 +0000 UTC m=+2808.354395325"
Oct 30 07:44:08.048348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.048312    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:08.065346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.064117    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:08.073129 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:08.073093    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9255ad61_2a79_4b36_972c_ed2dd6eb0b1b.slice/crio-7da6a7dba076d679ec8a5a58d2db4eaefecc2d5b0269964326938413f104965e WatchSource:0}: Error finding container 7da6a7dba076d679ec8a5a58d2db4eaefecc2d5b0269964326938413f104965e: Status 404 returned error can't find the container with id 7da6a7dba076d679ec8a5a58d2db4eaefecc2d5b0269964326938413f104965e
Oct 30 07:44:08.075195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.075157    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:44:08.187162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.187122    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\" (UniqueName: \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\") pod \"76ef30a5-18db-4bed-95a6-963facbcd2ea\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") "
Oct 30 07:44:08.187492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.187232    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\") pod \"76ef30a5-18db-4bed-95a6-963facbcd2ea\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") "
Oct 30 07:44:08.187492 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.187386    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\") pod \"76ef30a5-18db-4bed-95a6-963facbcd2ea\" (UID: \"76ef30a5-18db-4bed-95a6-963facbcd2ea\") "
Oct 30 07:44:08.188891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.188860    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca" pod "76ef30a5-18db-4bed-95a6-963facbcd2ea" (UID: "76ef30a5-18db-4bed-95a6-963facbcd2ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:08.190276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.190251    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca" pod "76ef30a5-18db-4bed-95a6-963facbcd2ea" (UID: "76ef30a5-18db-4bed-95a6-963facbcd2ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:44:08.190683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.190637    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk" pod "76ef30a5-18db-4bed-95a6-963facbcd2ea" (UID: "76ef30a5-18db-4bed-95a6-963facbcd2ea"). InnerVolumeSpecName "kube-api-access-z8wkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:08.290261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.290218    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wkk\" (UniqueName: \"kubernetes.io/projected/76ef30a5-18db-4bed-95a6-963facbcd2ea-kube-api-access-z8wkk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:08.290261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.290262    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:08.290522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.290279    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ef30a5-18db-4bed-95a6-963facbcd2ea-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:08.870620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.870544    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"
Oct 30 07:44:08.871064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.870553    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm" event={"ID":"76ef30a5-18db-4bed-95a6-963facbcd2ea","Type":"ContainerDied","Data":"a87d4205326fa46e73037af7088ac3092e86e5db66db976f7c8c9f558a6711e6"}
Oct 30 07:44:08.871064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.870668    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87d4205326fa46e73037af7088ac3092e86e5db66db976f7c8c9f558a6711e6"
Oct 30 07:44:08.872901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.872866    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" event={"ID":"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b","Type":"ContainerStarted","Data":"a77ef8eaafd87a5039d3486f9611503fa328f2542e0baefae7a39e4aeec67a28"}
Oct 30 07:44:08.873073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.872920    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" event={"ID":"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b","Type":"ContainerStarted","Data":"7da6a7dba076d679ec8a5a58d2db4eaefecc2d5b0269964326938413f104965e"}
Oct 30 07:44:08.918697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:08.918647    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" podStartSLOduration=1.918633422 podStartE2EDuration="1.918633422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:08.916213621 +0000 UTC m=+2809.348977530" watchObservedRunningTime="2026-10-30 07:44:08.918633422 +0000 UTC m=+2809.351397314"
Oct 30 07:44:09.883494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:09.883459    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:09.951894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:09.951852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:10.893873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.893832    6738 generic.go:356] "Generic (PLEG): container finished" podID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerID="ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4" exitCode=0
Oct 30 07:44:10.894678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.893902    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerDied","Data":"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"}
Oct 30 07:44:10.894678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.894472    6738 scope.go:122] "RemoveContainer" containerID="ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"
Oct 30 07:44:10.896507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.896471    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:10.896606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.896549    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:10.896606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:10.896585    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:11.877719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:11.868233    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:11.948010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:11.947970    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerStarted","Data":"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"}
Oct 30 07:44:12.889843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:12.889807    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:44:12.892362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:12.891403    6738 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused" start-of-body=
Oct 30 07:44:12.892362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:12.891461    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused"
Oct 30 07:44:13.838948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:13.838909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:13.902422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:13.896494    6738 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused" start-of-body=
Oct 30 07:44:13.902422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:13.896569    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused"
Oct 30 07:44:14.887320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:14.874803    6738 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused" start-of-body=
Oct 30 07:44:14.887320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:14.874887    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused"
Oct 30 07:44:15.101307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.101258    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"]
Oct 30 07:44:15.102518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.102492    6738 cpu_manager.go:414] "containerMap: removing container" podUID="76ef30a5-18db-4bed-95a6-963facbcd2ea" containerName="revoke-keys"
Oct 30 07:44:15.102518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.102519    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef30a5-18db-4bed-95a6-963facbcd2ea" containerName="revoke-keys"
Oct 30 07:44:15.102731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.102719    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef30a5-18db-4bed-95a6-963facbcd2ea" containerName="revoke-keys"
Oct 30 07:44:15.105911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.105894    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.123126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.123094    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"]
Oct 30 07:44:15.128974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.128940    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:44:15.129193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.128943    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:44:15.133219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.133185    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvg4d\" (UniqueName: \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.133402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.133372    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.133486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.133446    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.235088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.234983    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.235283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.235107    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.235428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.235332    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvg4d\" (UniqueName: \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.235859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.235839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.237596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.237561    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.294802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.294767    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvg4d\" (UniqueName: \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\") pod \"maas-api-revoke-sub-faac18c2-ddlh2\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.424909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.424877    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:15.804005 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:15.803964    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6484ff4_e6fd_4f37_883c_0d0a8d3e391e.slice/crio-86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c WatchSource:0}: Error finding container 86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c: Status 404 returned error can't find the container with id 86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c
Oct 30 07:44:15.804194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.804143    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"]
Oct 30 07:44:15.887318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.884639    6738 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused" start-of-body=
Oct 30 07:44:15.887318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.884707    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused"
Oct 30 07:44:15.965573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:15.963401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:16.025341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:16.024632    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2" event={"ID":"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e","Type":"ContainerStarted","Data":"86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c"}
Oct 30 07:44:16.892652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:16.892435    6738 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused" start-of-body=
Oct 30 07:44:16.892652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:16.892508    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.111:15021/healthz/ready\": dial tcp 10.133.1.111:15021: connect: connection refused"
Oct 30 07:44:17.034883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.034851    6738 generic.go:356] "Generic (PLEG): container finished" podID="f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" containerID="d8c64d6cac4e10eaef723783ae59195116c04875f4c9c8fd8f3ee7c076de5940" exitCode=0
Oct 30 07:44:17.035072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.034979    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2" event={"ID":"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e","Type":"ContainerDied","Data":"d8c64d6cac4e10eaef723783ae59195116c04875f4c9c8fd8f3ee7c076de5940"}
Oct 30 07:44:17.831687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.831648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:17.937199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.935828    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:44:17.937199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.936870    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:44:17.938067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:17.938041    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:44:18.235484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.235458    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:18.313564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.313531    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\") pod \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") "
Oct 30 07:44:18.313773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.313668    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\") pod \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") "
Oct 30 07:44:18.313773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.313760    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\" (UniqueName: \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\") pod \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\" (UID: \"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e\") "
Oct 30 07:44:18.314020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.313991    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca" pod "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" (UID: "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:18.314545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.314523    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:18.315809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.315790    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca" pod "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" (UID: "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:44:18.315953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.315934    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d" pod "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" (UID: "f6484ff4-e6fd-4f37-883c-0d0a8d3e391e"). InnerVolumeSpecName "kube-api-access-xvg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:18.416128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.416087    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:18.416128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.416120    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvg4d\" (UniqueName: \"kubernetes.io/projected/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e-kube-api-access-xvg4d\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:18.875619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:18.875592    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:44:19.062329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:19.062249    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2" event={"ID":"f6484ff4-e6fd-4f37-883c-0d0a8d3e391e","Type":"ContainerDied","Data":"86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c"}
Oct 30 07:44:19.063221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:19.063172    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"
Oct 30 07:44:19.063526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:19.063109    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86b500223bab8ad354330dc9a20577e29f42a15f3a60b8ce4441e65b92ac4d1c"
Oct 30 07:44:19.721795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:19.721761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:20.898061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:20.898028    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:44:21.651600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:21.651562    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:23.498317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:23.484172    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:25.031903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:25.031867    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"]
Oct 30 07:44:25.044097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:25.044064    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab411df-97q6t"]
Oct 30 07:44:25.268539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:25.268498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:25.692429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:25.692395    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1980d93-9c03-48f9-acbc-f1d36e3b8e3e" path="/var/lib/kubelet/pods/b1980d93-9c03-48f9-acbc-f1d36e3b8e3e/volumes"
Oct 30 07:44:27.042497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:27.042456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:27.232140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:27.232106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:27.349547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:27.349457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:28.662896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.662861    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"]
Oct 30 07:44:28.663776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.663760    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" containerName="revoke-keys"
Oct 30 07:44:28.663858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.663777    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" containerName="revoke-keys"
Oct 30 07:44:28.663914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.663905    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" containerName="revoke-keys"
Oct 30 07:44:28.667635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.667611    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.688602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.688570    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"]
Oct 30 07:44:28.743245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.743208    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvd2\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.743590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.743560    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.743740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.743687    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.743874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.743759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.785941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.785888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:28.845479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.845441    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.845697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.845506    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.845697 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:28.845597    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c1b5d7/agc-cred-ad217a913a5d: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:28.845697 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:28.845617    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:28.845697 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:28.845683    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials podName:0012979c-04e7-45a8-9c20-763d84436f04 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:29.345666858 +0000 UTC m=+2829.778430730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww" (UID: "0012979c-04e7-45a8-9c20-763d84436f04") : secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:28.845895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.845801    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvd2\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.845895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.845864    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.846364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.846338    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.846680 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.846659    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:28.916375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.911852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:28.916375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:28.914315    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvd2\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:29.212853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.212739    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:44:29.292090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.292049    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:29.360434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.360397    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:29.360600 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:29.360573    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c1b5d7/agc-cred-ad217a913a5d: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:29.360600 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:29.360595    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:29.360694 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:29.360665    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials podName:0012979c-04e7-45a8-9c20-763d84436f04 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:30.360644825 +0000 UTC m=+2830.793408702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww" (UID: "0012979c-04e7-45a8-9c20-763d84436f04") : secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:29.422166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.422129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:29.537685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.537583    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:29.613318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.612712    6738 scope.go:122] "RemoveContainer" containerID="fbacba39f877503f88c6d80105599b40c2ddc3c2b254fe123ba391d7550cde5a"
Oct 30 07:44:29.760612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.760581    6738 scope.go:122] "RemoveContainer" containerID="22052d59e3e502e7ca7cc390ba37bf62bb4ffa2692b600e150f027387eef5ff0"
Oct 30 07:44:29.827382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.827343    6738 scope.go:122] "RemoveContainer" containerID="2866d7051c1da4c420f0f6a77257388fefd38e64088cb55e7e2c218d1f09c381"
Oct 30 07:44:29.841129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:29.841100    6738 scope.go:122] "RemoveContainer" containerID="34d551980dd89c90ea6cd851e44a6f2af79735e977d12ae156eb3d047480901a"
Oct 30 07:44:30.163040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.163002    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:44:30.175195 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.175162    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"]
Oct 30 07:44:30.176122 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:30.176084    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww" podUID="0012979c-04e7-45a8-9c20-763d84436f04"
Oct 30 07:44:30.222712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.222678    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:30.232795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.232767    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:30.246483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.246455    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:30.305979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.305948    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\") pod \"0012979c-04e7-45a8-9c20-763d84436f04\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") "
Oct 30 07:44:30.306203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306036    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\") pod \"0012979c-04e7-45a8-9c20-763d84436f04\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") "
Oct 30 07:44:30.306307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306207    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\") pod \"0012979c-04e7-45a8-9c20-763d84436f04\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") "
Oct 30 07:44:30.306438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306412    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume" pod "0012979c-04e7-45a8-9c20-763d84436f04" (UID: "0012979c-04e7-45a8-9c20-763d84436f04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:30.306719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306681    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume" pod "0012979c-04e7-45a8-9c20-763d84436f04" (UID: "0012979c-04e7-45a8-9c20-763d84436f04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:30.306940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306920    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:30.307023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.306943    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0012979c-04e7-45a8-9c20-763d84436f04-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:30.308502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.308479    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2" pod "0012979c-04e7-45a8-9c20-763d84436f04" (UID: "0012979c-04e7-45a8-9c20-763d84436f04"). InnerVolumeSpecName "kube-api-access-zwvd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:30.408203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.408160    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww\" (UID: \"0012979c-04e7-45a8-9c20-763d84436f04\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:30.408400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:30.408321    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvd2\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-kube-api-access-zwvd2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:30.408400 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:30.408349    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c1b5d7/agc-cred-ad217a913a5d: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:30.408400 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:30.408369    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww: secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:30.408521 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:30.408425    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials podName:0012979c-04e7-45a8-9c20-763d84436f04 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:32.408409087 +0000 UTC m=+2832.841172961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww" (UID: "0012979c-04e7-45a8-9c20-763d84436f04") : secret "agc-cred-ad217a913a5d" not found
Oct 30 07:44:31.232323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.232265    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"
Oct 30 07:44:31.320173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.320124    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"]
Oct 30 07:44:31.325451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.325414    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-84675sfww"]
Oct 30 07:44:31.432830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.432785    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0012979c-04e7-45a8-9c20-763d84436f04-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:31.554607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.554523    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:31.693690 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:31.693656    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0012979c-04e7-45a8-9c20-763d84436f04" path="/var/lib/kubelet/pods/0012979c-04e7-45a8-9c20-763d84436f04/volumes"
Oct 30 07:44:33.322165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:33.322129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:33.462136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:33.462092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:34.213808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.213751    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" podUID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" containerName="payload-processing" containerID="cri-o://4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434" gracePeriod=25
Oct 30 07:44:34.520458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.520428    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:44:34.592157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.592112    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\") pod \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") "
Oct 30 07:44:34.592409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.592386    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\") pod \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") "
Oct 30 07:44:34.592539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.592520    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\") pod \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") "
Oct 30 07:44:34.592600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.592575    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\") pod \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\" (UID: \"88dd0509-5d2d-40ea-bb97-8e940a462bc2\") "
Oct 30 07:44:34.593371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.593001    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume" pod "88dd0509-5d2d-40ea-bb97-8e940a462bc2" (UID: "88dd0509-5d2d-40ea-bb97-8e940a462bc2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:34.593371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.593338    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume" pod "88dd0509-5d2d-40ea-bb97-8e940a462bc2" (UID: "88dd0509-5d2d-40ea-bb97-8e940a462bc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:34.593574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.593451    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:34.595441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.595411    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials" pod "88dd0509-5d2d-40ea-bb97-8e940a462bc2" (UID: "88dd0509-5d2d-40ea-bb97-8e940a462bc2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:34.596002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.595978    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj" pod "88dd0509-5d2d-40ea-bb97-8e940a462bc2" (UID: "88dd0509-5d2d-40ea-bb97-8e940a462bc2"). InnerVolumeSpecName "kube-api-access-wjqbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:34.694929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.694894    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqbj\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-kube-api-access-wjqbj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:34.694929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.694923    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88dd0509-5d2d-40ea-bb97-8e940a462bc2-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:34.694929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:34.694934    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88dd0509-5d2d-40ea-bb97-8e940a462bc2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:35.164234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.164190    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" podUID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" containerName="payload-processing" containerID="cri-o://e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69" gracePeriod=25
Oct 30 07:44:35.244074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.244041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:35.267083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.267049    6738 generic.go:356] "Generic (PLEG): container finished" podID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" containerID="4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434" exitCode=0
Oct 30 07:44:35.267283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.267116    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" event={"ID":"88dd0509-5d2d-40ea-bb97-8e940a462bc2","Type":"ContainerDied","Data":"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"}
Oct 30 07:44:35.267283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.267128    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"
Oct 30 07:44:35.267283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.267222    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2" event={"ID":"88dd0509-5d2d-40ea-bb97-8e940a462bc2","Type":"ContainerDied","Data":"46d4fe2ef879e45aae629796b7b024250af151402cb28a4e8a27601516a2ef48"}
Oct 30 07:44:35.267283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.267254    6738 scope.go:122] "RemoveContainer" containerID="4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"
Oct 30 07:44:35.279959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.279936    6738 scope.go:122] "RemoveContainer" containerID="4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"
Oct 30 07:44:35.280307 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:35.280263    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434\": container with ID starting with 4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434 not found: ID does not exist" containerID="4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"
Oct 30 07:44:35.280420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.280327    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434"} err="failed to get container status \"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434\": rpc error: code = NotFound desc = could not find container \"4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434\": container with ID starting with 4ad3a7ce808678409ae47cb91a5deaef6d12167dc59bf4423c34fe7fb39e8434 not found: ID does not exist"
Oct 30 07:44:35.350544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.350502    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:44:35.355093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.355057    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6bcbsk4g2"]
Oct 30 07:44:35.529568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.529538    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:44:35.631574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.631530    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\") pod \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") "
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.631700    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\") pod \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") "
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.631830    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\") pod \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") "
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.631893    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\") pod \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\" (UID: \"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7\") "
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.632544    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume" pod "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" (UID: "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.632863    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:35.633942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.633844    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume" pod "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" (UID: "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:35.634726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.634703    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials" pod "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" (UID: "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:35.635112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.635091    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8" pod "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" (UID: "543082cc-cf2a-47dc-9c1c-e8fa234ad3b7"). InnerVolumeSpecName "kube-api-access-xh7n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:35.671416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.671381    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:35.692500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.692471    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" path="/var/lib/kubelet/pods/88dd0509-5d2d-40ea-bb97-8e940a462bc2/volumes"
Oct 30 07:44:35.739241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.739198    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7n8\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-kube-api-access-xh7n8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:35.739241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.739244    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:35.739547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.739261    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:44:35.821463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:35.821348    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:36.214041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.214009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:36.280414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.280370    6738 generic.go:356] "Generic (PLEG): container finished" podID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" containerID="e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69" exitCode=0
Oct 30 07:44:36.280414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.280409    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"
Oct 30 07:44:36.280713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.280439    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" event={"ID":"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7","Type":"ContainerDied","Data":"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"}
Oct 30 07:44:36.280713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.280490    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2" event={"ID":"543082cc-cf2a-47dc-9c1c-e8fa234ad3b7","Type":"ContainerDied","Data":"b9b0c59b5bf25c699bd68be251ac26b0a78e532af1a238ca34f45db756e8a2ad"}
Oct 30 07:44:36.280713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.280512    6738 scope.go:122] "RemoveContainer" containerID="e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"
Oct 30 07:44:36.293038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.293008    6738 scope.go:122] "RemoveContainer" containerID="e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"
Oct 30 07:44:36.293428 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:36.293398    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69\": container with ID starting with e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69 not found: ID does not exist" containerID="e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"
Oct 30 07:44:36.293527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.293437    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69"} err="failed to get container status \"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69\": rpc error: code = NotFound desc = could not find container \"e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69\": container with ID starting with e058304412611d16bdfa7f5d6b3755703f5280b2a360311acde5d5bf6be69f69 not found: ID does not exist"
Oct 30 07:44:36.339070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.339036    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:44:36.351051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.351015    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-ffbfszwt2"]
Oct 30 07:44:36.735426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.735390    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:36.856753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:36.856694    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:37.010379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:37.003773    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:37.282221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:37.282120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:37.520748 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:44:37.520694    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:44:37.693250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:37.693215    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" path="/var/lib/kubelet/pods/543082cc-cf2a-47dc-9c1c-e8fa234ad3b7/volumes"
Oct 30 07:44:38.322111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:38.322064    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:38.448197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:38.448119    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:39.250230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:39.250182    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:39.400840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:39.400807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:39.824831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:39.824781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:40.499548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.488787    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:44:40.532469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.515846    6738 cpu_manager.go:414] "containerMap: removing container" podUID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.656333    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.656378    6738 cpu_manager.go:414] "containerMap: removing container" podUID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.656388    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.656723    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="543082cc-cf2a-47dc-9c1c-e8fa234ad3b7" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.656750    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="88dd0509-5d2d-40ea-bb97-8e940a462bc2" containerName="payload-processing"
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.534075    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:40.671430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.665271    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.688644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.681914    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f480c6-openshift-default-dockercfg-m97vg\""
Oct 30 07:44:40.688644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.685580    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:44:40.849762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.849726    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.850125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.850093    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.850375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.850350    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.850621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.850601    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.850799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.850778    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.850970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.850943    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.851348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.851325    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhqx\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.851500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.851477    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.851573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.851519    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.933570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.933479    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:40.952745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952706    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.952972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952770    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.952972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952802    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.952972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952858    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.952972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952888    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.952972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952914    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.952974    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhqx\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953067    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953097    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953169    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953473    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953534    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953546    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.953671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.953614    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.955592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.955568    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.957260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.957240    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.982055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.982025    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhqx\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:40.982205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:40.982031    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\") pod \"e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") " pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:41.018609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.018573    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:41.095851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.095813    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:41.673699 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.673605    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:44:41.678810 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:41.678775    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1b3a58_f2b6_4529_9989_234cfab4f441.slice/crio-86d5b2f4a56b5f173e2fe8345c9b10a7acc7e35a87d5845844a562567007e51f WatchSource:0}: Error finding container 86d5b2f4a56b5f173e2fe8345c9b10a7acc7e35a87d5845844a562567007e51f: Status 404 returned error can't find the container with id 86d5b2f4a56b5f173e2fe8345c9b10a7acc7e35a87d5845844a562567007e51f
Oct 30 07:44:41.685136 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.685091    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:41.685256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.685190    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:41.685256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:41.685233    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:44:42.128372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:42.128332    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:42.394885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:42.394789    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerStarted","Data":"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"}
Oct 30 07:44:42.394885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:42.394845    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerStarted","Data":"86d5b2f4a56b5f173e2fe8345c9b10a7acc7e35a87d5845844a562567007e51f"}
Oct 30 07:44:42.457022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:42.456964    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podStartSLOduration=2.456948526 podStartE2EDuration="2.456948526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:42.441090114 +0000 UTC m=+2842.873854009" watchObservedRunningTime="2026-10-30 07:44:42.456948526 +0000 UTC m=+2842.889712489"
Oct 30 07:44:43.019501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:43.019465    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:43.026443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:43.026414    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:43.418183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:43.418147    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:43.419201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:43.419180    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:44:44.232316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:44.223514    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:44.434321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:44.373488    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:45.165803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:45.165763    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:45.429777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:45.429661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:45.576719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:45.576680    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:47.111306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.111273    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:47.294003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.293959    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:47.391629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.391590    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:44:47.398166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.398120    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.414395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.414358    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:44:47.420249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.419123    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f480c6-dockercfg-hf8ms\""
Oct 30 07:44:47.420249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.419132    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f480c6\""
Oct 30 07:44:47.459440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.459387    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:47.488313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.484654    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.488313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.484864    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgsw\" (UniqueName: \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.638494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.638450    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:44:47.776429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.773151    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.794657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.794616    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgsw\" (UniqueName: \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.795154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.784796    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:47.795321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.778869    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.805410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.805365    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:44:47.821093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.821059    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgsw\" (UniqueName: \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\") pod \"payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:47.841239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.841206    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:47.937327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.918965    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:47.937327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:47.919150    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjngl\" (UniqueName: \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.018434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.018385    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:48.021154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.021122    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.021326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.021188    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjngl\" (UniqueName: \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.022009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.021939    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.047686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.047651    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjngl\" (UniqueName: \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\") pod \"payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.119121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.119088    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.332920 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:48.332889    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed16e67_8162_4070_acdf_ce03d7865879.slice/crio-11daa5805ca61a41b2ba5ed47ee237e71c4e342f8684b114f2801ca69eb0272e WatchSource:0}: Error finding container 11daa5805ca61a41b2ba5ed47ee237e71c4e342f8684b114f2801ca69eb0272e: Status 404 returned error can't find the container with id 11daa5805ca61a41b2ba5ed47ee237e71c4e342f8684b114f2801ca69eb0272e
Oct 30 07:44:48.334712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.334679    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:44:48.445374 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:44:48.445345    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab94f60_9bbc_4f4e_976b_58c9e401974e.slice/crio-fe7cef8ad1c3bed839b49db744a4a08bf679e8f33622a1763c16ad21a3564750 WatchSource:0}: Error finding container fe7cef8ad1c3bed839b49db744a4a08bf679e8f33622a1763c16ad21a3564750: Status 404 returned error can't find the container with id fe7cef8ad1c3bed839b49db744a4a08bf679e8f33622a1763c16ad21a3564750
Oct 30 07:44:48.446545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.446520    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:44:48.682217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.682179    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" event={"ID":"5ab94f60-9bbc-4f4e-976b-58c9e401974e","Type":"ContainerStarted","Data":"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"}
Oct 30 07:44:48.682462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.682237    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" event={"ID":"5ab94f60-9bbc-4f4e-976b-58c9e401974e","Type":"ContainerStarted","Data":"fe7cef8ad1c3bed839b49db744a4a08bf679e8f33622a1763c16ad21a3564750"}
Oct 30 07:44:48.682462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.682271    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:44:48.684751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.684699    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" event={"ID":"4ed16e67-8162-4070-acdf-ce03d7865879","Type":"ContainerStarted","Data":"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"}
Oct 30 07:44:48.684923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.684764    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" event={"ID":"4ed16e67-8162-4070-acdf-ce03d7865879","Type":"ContainerStarted","Data":"11daa5805ca61a41b2ba5ed47ee237e71c4e342f8684b114f2801ca69eb0272e"}
Oct 30 07:44:48.684923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.684791    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:48.727457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.727401    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" podStartSLOduration=1.727385119 podStartE2EDuration="1.727385119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:48.72392558 +0000 UTC m=+2849.156689472" watchObservedRunningTime="2026-10-30 07:44:48.727385119 +0000 UTC m=+2849.160149013"
Oct 30 07:44:48.767458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:48.767394    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" podStartSLOduration=1.76737785 podStartE2EDuration="1.76737785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:48.765361975 +0000 UTC m=+2849.198125867" watchObservedRunningTime="2026-10-30 07:44:48.76737785 +0000 UTC m=+2849.200141744"
Oct 30 07:44:49.344522 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:49.344488    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:49.863770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:49.863732    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:51.754053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:51.754020    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:53.659550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:53.659511    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:55.559154 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:55.559118    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:57.390120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:57.390079    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:59.174628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:59.174588    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:44:59.697634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:59.697605    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:44:59.697820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:44:59.697670    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:45:00.032684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.032597    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"]
Oct 30 07:45:00.036991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.036961    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155226b9-2tvhm"]
Oct 30 07:45:00.160502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.160464    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:00.170362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.170321    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.178724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.178693    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:00.192153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.192112    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:00.192348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.192242    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.199405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.199375    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:00.200345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.200314    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.200973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.200950    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:45:00.210469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.210438    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:00.211325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.211281    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.223777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.223744    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:00.223968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.223883    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.228063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.228035    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:00.233419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.233388    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:00.238718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.238680    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:00.245527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.245499    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:00.247879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.247843    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qc7\" (UniqueName: \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.248025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248000    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66g9l\" (UniqueName: \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.248131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248113    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.248335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248314    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8zm\" (UniqueName: \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.248605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248572    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8797\" (UniqueName: \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.248725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248648    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.248725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248686    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.249163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.248753    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.249870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.249830    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx86p\" (UniqueName: \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.250103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.250070    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.271526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.271479    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:00.271703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.271625    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.294029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.290483    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:00.294029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.290533    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:00.294029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.291701    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.296508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.296483    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:00.296715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.296693    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:00.296987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.296965    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.303409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.303382    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.317630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.317598    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:00.346032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.345998    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:00.353565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353524    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.353741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353606    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.353741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353653    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8zm\" (UniqueName: \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.353873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353774    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r72x\" (UniqueName: \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.353873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353802    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4nt\" (UniqueName: \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.353988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353848    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78qg\" (UniqueName: \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.353988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353933    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8797\" (UniqueName: \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.353988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.353966    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.354142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354021    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.354142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354063    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.354142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354126    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.354322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354156    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.354322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354182    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx86p\" (UniqueName: \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.354322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.354322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354267    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.354536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354359    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qc7\" (UniqueName: \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.354536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354448    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zwl\" (UniqueName: \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.354847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.354589    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66g9l\" (UniqueName: \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.356869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.356837    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.358090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.358059    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.358287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.358257    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.359251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.359222    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.361169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.361140    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.377800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.376841    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:00.377800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.376885    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:00.377800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.377229    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.394430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.391490    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:00.394430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.391620    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.401419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.399164    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:00.419447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.419370    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:00.425028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.424900    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8797\" (UniqueName: \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.426436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.425958    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx86p\" (UniqueName: \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.426827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.426804    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:00.428553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.428495    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66g9l\" (UniqueName: \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.433348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.433312    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qc7\" (UniqueName: \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.433911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.433887    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.441371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.437501    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8zm\" (UniqueName: \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.448255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.448052    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456488    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456537    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456580    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pfr\" (UniqueName: \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456640    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456667    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zwl\" (UniqueName: \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456719    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.456774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456779    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lx9\" (UniqueName: \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456821    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456926    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r72x\" (UniqueName: \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4nt\" (UniqueName: \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456972    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78qg\" (UniqueName: \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.456994    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.457015    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.457347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.457040    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v427g\" (UniqueName: \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.458001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.457969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.459358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.458596    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.459755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.459729    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.461016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.460990    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.465947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.465916    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:00.475273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.475240    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.475474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.475243    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:00.488974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.487576    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.494470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.491446    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:00.507367 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.500044    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:00.516370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.515101    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:00.516370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.515246    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.527359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.526427    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:00.536049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.536013    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zwl\" (UniqueName: \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.536797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.536722    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4nt\" (UniqueName: \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.536933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.536894    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:00.537010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.536935    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:00.537839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.537813    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r72x\" (UniqueName: \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.539508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.539440    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78qg\" (UniqueName: \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.548482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.542716    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:00.557288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.557241    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:00.558071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558045    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pfr\" (UniqueName: \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.558187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558127    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.558187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558174    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.558375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558233    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.558375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558264    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmm8\" (UniqueName: \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.558375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558342    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lx9\" (UniqueName: \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.558549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558372    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks2z\" (UniqueName: \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.558549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558423    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.558549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558503    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vf7h\" (UniqueName: \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.558713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.558594    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.561342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.561269    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.561456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.561360    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v427g\" (UniqueName: \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.561912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.559879    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.562859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.562833    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.562965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.560529    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.564560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.564530    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:00.606534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.604343    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pfr\" (UniqueName: \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.606534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.604900    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lx9\" (UniqueName: \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.612410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.610532    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v427g\" (UniqueName: \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.621923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.621893    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:00.644847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.644804    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:00.673181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.673137    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.673787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.673756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.674083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.674059    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmm8\" (UniqueName: \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.674449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.674419    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks2z\" (UniqueName: \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.674545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.674501    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.674679 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.674656    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf7h\" (UniqueName: \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.676251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.676007    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.676852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.676816    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.677260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.677242    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:00.681886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.681857    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.696940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.696903    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:00.705886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.705851    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf7h\" (UniqueName: \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\") pod \"maas-api-key-cleanup-29889105-ggxfv\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.708066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.708016    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks2z\" (UniqueName: \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:00.733443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.732117    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:00.759577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.759227    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmm8\" (UniqueName: \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\") pod \"maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:00.790358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.790192    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:00.822665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.822433    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:00.870360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.870038    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:00.947252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.947002    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:00.951819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:00.951779    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:01.003109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.003083    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:01.314621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.314475    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:01.364955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.364920    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:01.383893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.383799    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:01.479430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.479389    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:01.519890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.519858    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:01.589646 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:01.589613    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4e0dcf_0f6f_4ba7_8b7e_e3cd72150c1c.slice/crio-b1b658703fc33e15e45a4dd22e608f7f64c6b724f1b2917107f13a9ffe36a36f WatchSource:0}: Error finding container b1b658703fc33e15e45a4dd22e608f7f64c6b724f1b2917107f13a9ffe36a36f: Status 404 returned error can't find the container with id b1b658703fc33e15e45a4dd22e608f7f64c6b724f1b2917107f13a9ffe36a36f
Oct 30 07:45:01.593573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.593054    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:01.625032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.625003    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:01.669419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.669385    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:01.680194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.678413    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:01.684432 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:01.684259    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7256a39a_061a_418e_b299_e67eacf4f780.slice/crio-14101fe55f747f0618c92e92f17895ec2e7c9dad24ff11e1af5cfba27215dfc6 WatchSource:0}: Error finding container 14101fe55f747f0618c92e92f17895ec2e7c9dad24ff11e1af5cfba27215dfc6: Status 404 returned error can't find the container with id 14101fe55f747f0618c92e92f17895ec2e7c9dad24ff11e1af5cfba27215dfc6
Oct 30 07:45:01.709411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.709379    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7" path="/var/lib/kubelet/pods/9e853cf2-86a3-40b7-b5ad-5fc1c89f71f7/volumes"
Oct 30 07:45:01.709876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.709857    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:01.709938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.709907    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:01.709938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.709924    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:01.722868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.722818    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:01.729401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.729375    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:01.731142 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:01.731115    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6222c938_39d6_4d25_b97a_22be3e24e33b.slice/crio-d2472f2cd503327d496105665ca62eea96a0293ff5cc19a542885e50c0e70ee8 WatchSource:0}: Error finding container d2472f2cd503327d496105665ca62eea96a0293ff5cc19a542885e50c0e70ee8: Status 404 returned error can't find the container with id d2472f2cd503327d496105665ca62eea96a0293ff5cc19a542885e50c0e70ee8
Oct 30 07:45:01.733539 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:01.733506    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160ab89d_32a5_47bd_b0a6_7dbb5d83407b.slice/crio-4fb93ab602a4556c65b8e620dd7c9a168339bb3f2b7190f6fd1d307f6e7d417b WatchSource:0}: Error finding container 4fb93ab602a4556c65b8e620dd7c9a168339bb3f2b7190f6fd1d307f6e7d417b: Status 404 returned error can't find the container with id 4fb93ab602a4556c65b8e620dd7c9a168339bb3f2b7190f6fd1d307f6e7d417b
Oct 30 07:45:01.734182 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:01.734157    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e20dd7_eea4_4d0d_a447_6d4d8afed2ed.slice/crio-819131edf8c4e0961106f8c81aef76be4135a5ec740b520704853c25f20a70f4 WatchSource:0}: Error finding container 819131edf8c4e0961106f8c81aef76be4135a5ec740b520704853c25f20a70f4: Status 404 returned error can't find the container with id 819131edf8c4e0961106f8c81aef76be4135a5ec740b520704853c25f20a70f4
Oct 30 07:45:01.737119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.735408    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:01.857400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.857290    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerStarted","Data":"4fb93ab602a4556c65b8e620dd7c9a168339bb3f2b7190f6fd1d307f6e7d417b"}
Oct 30 07:45:01.858858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.858823    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerStarted","Data":"14611554f49d7bff37afc2f5b6296bf0dcaf2d6d8bf3379323848ebd04449b75"}
Oct 30 07:45:01.860332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.860289    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerStarted","Data":"819131edf8c4e0961106f8c81aef76be4135a5ec740b520704853c25f20a70f4"}
Oct 30 07:45:01.861909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.861883    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerStarted","Data":"f34daf2d68166a97e0370e8e8e3932e622ebe3bfd4fedbdb665f8a15f98aa64c"}
Oct 30 07:45:01.862010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.861931    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerStarted","Data":"642ab35dbc97c72849a1f33b55ef01715176bc30806d13f70820cf39b57412bd"}
Oct 30 07:45:01.863437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.863409    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerStarted","Data":"d2472f2cd503327d496105665ca62eea96a0293ff5cc19a542885e50c0e70ee8"}
Oct 30 07:45:01.864688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.864662    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerStarted","Data":"b1b658703fc33e15e45a4dd22e608f7f64c6b724f1b2917107f13a9ffe36a36f"}
Oct 30 07:45:01.865927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.865898    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerStarted","Data":"14101fe55f747f0618c92e92f17895ec2e7c9dad24ff11e1af5cfba27215dfc6"}
Oct 30 07:45:01.867169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.867144    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerStarted","Data":"4e1515791610dc0f71bebc72809d08a9598b687db8b4b94e3ecf8fdabf32867f"}
Oct 30 07:45:01.868930 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.868907    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerStarted","Data":"0e4ce98970d00c87a3c0f3c532a00710b4f15e668bd662cefaaa3a9bba9c6989"}
Oct 30 07:45:01.870484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.870456    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerStarted","Data":"893923bd1074a9d189ce8bee12a0ef02a2b319d0a039db00b1b11059814a88e4"}
Oct 30 07:45:01.873737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.873405    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerStarted","Data":"abc17da08a83da5e7dc94c29af8772ef61caf97b927d06e8468e845e7e0abc38"}
Oct 30 07:45:01.876604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.876329    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerStarted","Data":"874851ae4a9e567c64c83133c5d1c1317473b4e7dbb97826ffb315e7c9d49b2f"}
Oct 30 07:45:01.878139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.878101    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerStarted","Data":"e28f91115375ab196a137ffe836197683c2b1417ba8e2e76d8198945a752bf13"}
Oct 30 07:45:01.880036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.880013    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerStarted","Data":"1486db05bd2049a387143cca0a01a2525356ded692730349b2f0fe4f49d4d572"}
Oct 30 07:45:01.881644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.881619    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerStarted","Data":"9fd7a9f2379e43027da48ba78647d497fa3b47054b8f9fda891b5fa610c3fe41"}
Oct 30 07:45:01.916800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:01.916740    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" podStartSLOduration=1.91671785 podStartE2EDuration="1.91671785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:01.911414081 +0000 UTC m=+2862.344177975" watchObservedRunningTime="2026-10-30 07:45:01.91671785 +0000 UTC m=+2862.349481743"
Oct 30 07:45:02.898596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.898560    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerStarted","Data":"8da022ff5688bd8cffb6adb37b12e6a46ea9927bf71139f9419d5f32e84913ed"}
Oct 30 07:45:02.900404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.900372    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerStarted","Data":"27c111cea6a80cd407b09ea2d0fb268d47b1a88a660187a6df8736e8069f5356"}
Oct 30 07:45:02.902010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.901988    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerStarted","Data":"83226db011a88e8d2dd692b495d055c91957c5d1c3604dfe855ea96aea8a55c2"}
Oct 30 07:45:02.904002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.903974    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerStarted","Data":"4c0d3c8503869ad6a067241c1013aecd3c9b66edbaa196f4556674e040d6a49e"}
Oct 30 07:45:02.905843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.905818    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerStarted","Data":"bbda1399f815fb1eb43d2806f39afa32295697de5b54657c8303feb8fd1fdb62"}
Oct 30 07:45:02.907540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.907515    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerStarted","Data":"2fdb9c320c04b0786f89f77010749581adb52cfb0dd2e09851baf5670ebff3a7"}
Oct 30 07:45:02.909263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.909240    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerStarted","Data":"e3553f15e57d2fbdf595794488fda96e4fd6c5c8e4799d01a2352875f0594a45"}
Oct 30 07:45:02.911141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.911118    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerStarted","Data":"fdc5b7f8eab4b63ae8c4baf67c464cb820304d50209392c8cf6b7463427039a9"}
Oct 30 07:45:02.912774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.912753    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerStarted","Data":"0afef1b3b08518fe7c09f56ef36d77dc495090f70211d1e0a18b13b5204f842a"}
Oct 30 07:45:02.914585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.914563    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerStarted","Data":"7acd71c97f25974662a2599b5007eac0f54c1bfdda5af1542b1eafcc386f0ad0"}
Oct 30 07:45:02.916035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.916010    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerStarted","Data":"555ee9ee6e6ffeb6963a029b759ee47a2aa47a93170c04727107ef6fc64a70c5"}
Oct 30 07:45:02.919198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.919174    6738 generic.go:356] "Generic (PLEG): container finished" podID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerID="f34daf2d68166a97e0370e8e8e3932e622ebe3bfd4fedbdb665f8a15f98aa64c" exitCode=60
Oct 30 07:45:02.919332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.919238    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerDied","Data":"f34daf2d68166a97e0370e8e8e3932e622ebe3bfd4fedbdb665f8a15f98aa64c"}
Oct 30 07:45:02.919448 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.919435    6738 scope.go:122] "RemoveContainer" containerID="f34daf2d68166a97e0370e8e8e3932e622ebe3bfd4fedbdb665f8a15f98aa64c"
Oct 30 07:45:02.943826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.943779    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" podStartSLOduration=2.943762468 podStartE2EDuration="2.943762468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.938727782 +0000 UTC m=+2863.371491689" watchObservedRunningTime="2026-10-30 07:45:02.943762468 +0000 UTC m=+2863.376526360"
Oct 30 07:45:02.982999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:02.982948    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" podStartSLOduration=2.982931694 podStartE2EDuration="2.982931694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.981756926 +0000 UTC m=+2863.414520818" watchObservedRunningTime="2026-10-30 07:45:02.982931694 +0000 UTC m=+2863.415695587"
Oct 30 07:45:03.023647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.023597    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" podStartSLOduration=3.02358216 podStartE2EDuration="3.02358216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.021921227 +0000 UTC m=+2863.454685122" watchObservedRunningTime="2026-10-30 07:45:03.02358216 +0000 UTC m=+2863.456346112"
Oct 30 07:45:03.066383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.066262    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" podStartSLOduration=3.066245886 podStartE2EDuration="3.066245886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.061367323 +0000 UTC m=+2863.494131218" watchObservedRunningTime="2026-10-30 07:45:03.066245886 +0000 UTC m=+2863.499009779"
Oct 30 07:45:03.107712 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.107639    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" podStartSLOduration=3.107617912 podStartE2EDuration="3.107617912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.101424811 +0000 UTC m=+2863.534188704" watchObservedRunningTime="2026-10-30 07:45:03.107617912 +0000 UTC m=+2863.540381807"
Oct 30 07:45:03.151466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.151406    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" podStartSLOduration=3.151390025 podStartE2EDuration="3.151390025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.146480051 +0000 UTC m=+2863.579243948" watchObservedRunningTime="2026-10-30 07:45:03.151390025 +0000 UTC m=+2863.584153947"
Oct 30 07:45:03.239096 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.239035    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" podStartSLOduration=3.239018858 podStartE2EDuration="3.239018858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.231872721 +0000 UTC m=+2863.664636613" watchObservedRunningTime="2026-10-30 07:45:03.239018858 +0000 UTC m=+2863.671782760"
Oct 30 07:45:03.279001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.278947    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" podStartSLOduration=3.278925294 podStartE2EDuration="3.278925294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.27187821 +0000 UTC m=+2863.704642103" watchObservedRunningTime="2026-10-30 07:45:03.278925294 +0000 UTC m=+2863.711689192"
Oct 30 07:45:03.335458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.331022    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" podStartSLOduration=3.330998415 podStartE2EDuration="3.330998415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.312424859 +0000 UTC m=+2863.745188751" watchObservedRunningTime="2026-10-30 07:45:03.330998415 +0000 UTC m=+2863.763762309"
Oct 30 07:45:03.365463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.365407    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" podStartSLOduration=3.365393328 podStartE2EDuration="3.365393328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.363694137 +0000 UTC m=+2863.796458030" watchObservedRunningTime="2026-10-30 07:45:03.365393328 +0000 UTC m=+2863.798157220"
Oct 30 07:45:03.405461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.405395    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" podStartSLOduration=3.405376255 podStartE2EDuration="3.405376255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.404264105 +0000 UTC m=+2863.837027998" watchObservedRunningTime="2026-10-30 07:45:03.405376255 +0000 UTC m=+2863.838140149"
Oct 30 07:45:03.462060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.461995    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" podStartSLOduration=3.461977524 podStartE2EDuration="3.461977524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.459535333 +0000 UTC m=+2863.892299231" watchObservedRunningTime="2026-10-30 07:45:03.461977524 +0000 UTC m=+2863.894741418"
Oct 30 07:45:03.475186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.475149    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:03.931262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.931225    6738 generic.go:356] "Generic (PLEG): container finished" podID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerID="2fdb9c320c04b0786f89f77010749581adb52cfb0dd2e09851baf5670ebff3a7" exitCode=60
Oct 30 07:45:03.931748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.931342    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerDied","Data":"2fdb9c320c04b0786f89f77010749581adb52cfb0dd2e09851baf5670ebff3a7"}
Oct 30 07:45:03.931748 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.931683    6738 scope.go:122] "RemoveContainer" containerID="2fdb9c320c04b0786f89f77010749581adb52cfb0dd2e09851baf5670ebff3a7"
Oct 30 07:45:03.933268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.933245    6738 generic.go:356] "Generic (PLEG): container finished" podID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerID="e3553f15e57d2fbdf595794488fda96e4fd6c5c8e4799d01a2352875f0594a45" exitCode=60
Oct 30 07:45:03.933412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.933326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerDied","Data":"e3553f15e57d2fbdf595794488fda96e4fd6c5c8e4799d01a2352875f0594a45"}
Oct 30 07:45:03.933675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.933656    6738 scope.go:122] "RemoveContainer" containerID="e3553f15e57d2fbdf595794488fda96e4fd6c5c8e4799d01a2352875f0594a45"
Oct 30 07:45:03.935638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.935619    6738 generic.go:356] "Generic (PLEG): container finished" podID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerID="fdc5b7f8eab4b63ae8c4baf67c464cb820304d50209392c8cf6b7463427039a9" exitCode=60
Oct 30 07:45:03.935763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.935736    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerDied","Data":"fdc5b7f8eab4b63ae8c4baf67c464cb820304d50209392c8cf6b7463427039a9"}
Oct 30 07:45:03.936018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.935968    6738 scope.go:122] "RemoveContainer" containerID="fdc5b7f8eab4b63ae8c4baf67c464cb820304d50209392c8cf6b7463427039a9"
Oct 30 07:45:03.937723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.937701    6738 generic.go:356] "Generic (PLEG): container finished" podID="26cab399-d2e1-4535-95a8-af6877be8c00" containerID="0afef1b3b08518fe7c09f56ef36d77dc495090f70211d1e0a18b13b5204f842a" exitCode=60
Oct 30 07:45:03.937769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.937726    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerDied","Data":"0afef1b3b08518fe7c09f56ef36d77dc495090f70211d1e0a18b13b5204f842a"}
Oct 30 07:45:03.937995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.937981    6738 scope.go:122] "RemoveContainer" containerID="0afef1b3b08518fe7c09f56ef36d77dc495090f70211d1e0a18b13b5204f842a"
Oct 30 07:45:03.939602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.939583    6738 generic.go:356] "Generic (PLEG): container finished" podID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerID="7acd71c97f25974662a2599b5007eac0f54c1bfdda5af1542b1eafcc386f0ad0" exitCode=60
Oct 30 07:45:03.939701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.939669    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerDied","Data":"7acd71c97f25974662a2599b5007eac0f54c1bfdda5af1542b1eafcc386f0ad0"}
Oct 30 07:45:03.939871 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.939855    6738 scope.go:122] "RemoveContainer" containerID="7acd71c97f25974662a2599b5007eac0f54c1bfdda5af1542b1eafcc386f0ad0"
Oct 30 07:45:03.941356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.941332    6738 generic.go:356] "Generic (PLEG): container finished" podID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerID="555ee9ee6e6ffeb6963a029b759ee47a2aa47a93170c04727107ef6fc64a70c5" exitCode=60
Oct 30 07:45:03.941442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.941407    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerDied","Data":"555ee9ee6e6ffeb6963a029b759ee47a2aa47a93170c04727107ef6fc64a70c5"}
Oct 30 07:45:03.941592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.941579    6738 scope.go:122] "RemoveContainer" containerID="555ee9ee6e6ffeb6963a029b759ee47a2aa47a93170c04727107ef6fc64a70c5"
Oct 30 07:45:03.943167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.943148    6738 generic.go:356] "Generic (PLEG): container finished" podID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerID="3ebe31ff019ae2bdb59a6c0cabc21a9355fee9932a793611188308103317cade" exitCode=60
Oct 30 07:45:03.943240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.943215    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerDied","Data":"3ebe31ff019ae2bdb59a6c0cabc21a9355fee9932a793611188308103317cade"}
Oct 30 07:45:03.943576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.943551    6738 scope.go:122] "RemoveContainer" containerID="3ebe31ff019ae2bdb59a6c0cabc21a9355fee9932a793611188308103317cade"
Oct 30 07:45:03.945140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.945116    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerStarted","Data":"e9257ff8f1dbdcf4af54b74814a41dfef85709ab178d1305f4a90fd18935f675"}
Oct 30 07:45:03.951171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.951137    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerStarted","Data":"c814ee97b8c14865ef40427187cb834d9c7cfc5ae75477633db836f9774f4f26"}
Oct 30 07:45:03.953140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.953111    6738 generic.go:356] "Generic (PLEG): container finished" podID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerID="8da022ff5688bd8cffb6adb37b12e6a46ea9927bf71139f9419d5f32e84913ed" exitCode=60
Oct 30 07:45:03.953253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.953229    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerDied","Data":"8da022ff5688bd8cffb6adb37b12e6a46ea9927bf71139f9419d5f32e84913ed"}
Oct 30 07:45:03.953528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.953509    6738 scope.go:122] "RemoveContainer" containerID="8da022ff5688bd8cffb6adb37b12e6a46ea9927bf71139f9419d5f32e84913ed"
Oct 30 07:45:03.956117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.956093    6738 generic.go:356] "Generic (PLEG): container finished" podID="7256a39a-061a-418e-b299-e67eacf4f780" containerID="27c111cea6a80cd407b09ea2d0fb268d47b1a88a660187a6df8736e8069f5356" exitCode=60
Oct 30 07:45:03.956441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.956389    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerDied","Data":"27c111cea6a80cd407b09ea2d0fb268d47b1a88a660187a6df8736e8069f5356"}
Oct 30 07:45:03.956711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.956493    6738 scope.go:122] "RemoveContainer" containerID="27c111cea6a80cd407b09ea2d0fb268d47b1a88a660187a6df8736e8069f5356"
Oct 30 07:45:03.958424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.958399    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerID="83226db011a88e8d2dd692b495d055c91957c5d1c3604dfe855ea96aea8a55c2" exitCode=60
Oct 30 07:45:03.958543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.958490    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerDied","Data":"83226db011a88e8d2dd692b495d055c91957c5d1c3604dfe855ea96aea8a55c2"}
Oct 30 07:45:03.958864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.958778    6738 scope.go:122] "RemoveContainer" containerID="83226db011a88e8d2dd692b495d055c91957c5d1c3604dfe855ea96aea8a55c2"
Oct 30 07:45:03.960458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.960436    6738 generic.go:356] "Generic (PLEG): container finished" podID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerID="4c0d3c8503869ad6a067241c1013aecd3c9b66edbaa196f4556674e040d6a49e" exitCode=60
Oct 30 07:45:03.960613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.960590    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerDied","Data":"4c0d3c8503869ad6a067241c1013aecd3c9b66edbaa196f4556674e040d6a49e"}
Oct 30 07:45:03.960768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.960755    6738 scope.go:122] "RemoveContainer" containerID="4c0d3c8503869ad6a067241c1013aecd3c9b66edbaa196f4556674e040d6a49e"
Oct 30 07:45:03.962351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.962284    6738 generic.go:356] "Generic (PLEG): container finished" podID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerID="bbda1399f815fb1eb43d2806f39afa32295697de5b54657c8303feb8fd1fdb62" exitCode=60
Oct 30 07:45:03.962435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.962382    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerDied","Data":"bbda1399f815fb1eb43d2806f39afa32295697de5b54657c8303feb8fd1fdb62"}
Oct 30 07:45:03.962614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:03.962600    6738 scope.go:122] "RemoveContainer" containerID="bbda1399f815fb1eb43d2806f39afa32295697de5b54657c8303feb8fd1fdb62"
Oct 30 07:45:04.105784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.105729    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" podStartSLOduration=4.105710116 podStartE2EDuration="4.105710116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.104474743 +0000 UTC m=+2864.537238640" watchObservedRunningTime="2026-10-30 07:45:04.105710116 +0000 UTC m=+2864.538474008"
Oct 30 07:45:04.153280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.153228    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:04.440804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.440690    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" podStartSLOduration=4.440674799 podStartE2EDuration="4.440674799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.438410576 +0000 UTC m=+2864.871174472" watchObservedRunningTime="2026-10-30 07:45:04.440674799 +0000 UTC m=+2864.873438692"
Oct 30 07:45:04.985054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.985021    6738 generic.go:356] "Generic (PLEG): container finished" podID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerID="e9257ff8f1dbdcf4af54b74814a41dfef85709ab178d1305f4a90fd18935f675" exitCode=60
Oct 30 07:45:04.985530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.985444    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerDied","Data":"e9257ff8f1dbdcf4af54b74814a41dfef85709ab178d1305f4a90fd18935f675"}
Oct 30 07:45:04.986804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.986766    6738 scope.go:122] "RemoveContainer" containerID="e9257ff8f1dbdcf4af54b74814a41dfef85709ab178d1305f4a90fd18935f675"
Oct 30 07:45:04.989807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.989467    6738 generic.go:356] "Generic (PLEG): container finished" podID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerID="b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3" exitCode=60
Oct 30 07:45:04.989807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.989559    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerDied","Data":"b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3"}
Oct 30 07:45:04.989807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.989604    6738 scope.go:122] "RemoveContainer" containerID="f34daf2d68166a97e0370e8e8e3932e622ebe3bfd4fedbdb665f8a15f98aa64c"
Oct 30 07:45:04.989975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.989908    6738 scope.go:122] "RemoveContainer" containerID="b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3"
Oct 30 07:45:04.990412 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:04.990356    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl_odh-ai-gateway-infra(3b12e79b-7d3f-4ee9-96c6-4d6d8533d299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299"
Oct 30 07:45:04.994043 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.994020    6738 generic.go:356] "Generic (PLEG): container finished" podID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerID="c814ee97b8c14865ef40427187cb834d9c7cfc5ae75477633db836f9774f4f26" exitCode=60
Oct 30 07:45:04.994165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.994108    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerDied","Data":"c814ee97b8c14865ef40427187cb834d9c7cfc5ae75477633db836f9774f4f26"}
Oct 30 07:45:04.994558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.994538    6738 scope.go:122] "RemoveContainer" containerID="c814ee97b8c14865ef40427187cb834d9c7cfc5ae75477633db836f9774f4f26"
Oct 30 07:45:04.997081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.997054    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerStarted","Data":"ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"}
Oct 30 07:45:04.999835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:04.999809    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerStarted","Data":"14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"}
Oct 30 07:45:05.002132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:05.002109    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerStarted","Data":"2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"}
Oct 30 07:45:05.004538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:05.004511    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerStarted","Data":"f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"}
Oct 30 07:45:05.006768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:05.006742    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerStarted","Data":"c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"}
Oct 30 07:45:05.410781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:05.410742    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:06.019687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.019643    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerStarted","Data":"371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"}
Oct 30 07:45:06.021920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.021891    6738 scope.go:122] "RemoveContainer" containerID="b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3"
Oct 30 07:45:06.022311 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.022271    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl_odh-ai-gateway-infra(3b12e79b-7d3f-4ee9-96c6-4d6d8533d299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299"
Oct 30 07:45:06.023743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.023722    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerStarted","Data":"d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"}
Oct 30 07:45:06.025739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.025716    6738 generic.go:356] "Generic (PLEG): container finished" podID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerID="ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49" exitCode=60
Oct 30 07:45:06.025856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.025788    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerDied","Data":"ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"}
Oct 30 07:45:06.025856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.025839    6738 scope.go:122] "RemoveContainer" containerID="8da022ff5688bd8cffb6adb37b12e6a46ea9927bf71139f9419d5f32e84913ed"
Oct 30 07:45:06.026091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.026073    6738 scope.go:122] "RemoveContainer" containerID="ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"
Oct 30 07:45:06.026528 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.026502    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8_odh-ai-gateway-infra(3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c"
Oct 30 07:45:06.027897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.027870    6738 generic.go:356] "Generic (PLEG): container finished" podID="7256a39a-061a-418e-b299-e67eacf4f780" containerID="7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff" exitCode=60
Oct 30 07:45:06.027999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.027961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerDied","Data":"7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff"}
Oct 30 07:45:06.028338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.028319    6738 scope.go:122] "RemoveContainer" containerID="7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff"
Oct 30 07:45:06.028674 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.028651    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k_odh-ai-gateway-infra(7256a39a-061a-418e-b299-e67eacf4f780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" podUID="7256a39a-061a-418e-b299-e67eacf4f780"
Oct 30 07:45:06.029947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.029928    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerID="4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf" exitCode=60
Oct 30 07:45:06.030021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.029987    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerDied","Data":"4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf"}
Oct 30 07:45:06.030234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.030220    6738 scope.go:122] "RemoveContainer" containerID="4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf"
Oct 30 07:45:06.030569 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.030547    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9_odh-ai-gateway-infra(f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f"
Oct 30 07:45:06.031971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.031949    6738 generic.go:356] "Generic (PLEG): container finished" podID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerID="15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d" exitCode=60
Oct 30 07:45:06.032092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.032069    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerDied","Data":"15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d"}
Oct 30 07:45:06.032250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.032237    6738 scope.go:122] "RemoveContainer" containerID="15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d"
Oct 30 07:45:06.032550 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.032527    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw_odh-ai-gateway-infra(09e994a2-e628-419c-b0dd-58e677cdf9fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc"
Oct 30 07:45:06.034143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.034123    6738 generic.go:356] "Generic (PLEG): container finished" podID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerID="14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f" exitCode=60
Oct 30 07:45:06.034237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.034203    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerDied","Data":"14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"}
Oct 30 07:45:06.034432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.034410    6738 scope.go:122] "RemoveContainer" containerID="14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"
Oct 30 07:45:06.034714 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.034692    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk_odh-ai-gateway-infra(5096ae27-38bc-4d4e-ad83-1efcaf982889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889"
Oct 30 07:45:06.036204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.036185    6738 generic.go:356] "Generic (PLEG): container finished" podID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerID="2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5" exitCode=60
Oct 30 07:45:06.036312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.036266    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerDied","Data":"2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"}
Oct 30 07:45:06.036489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.036474    6738 scope.go:122] "RemoveContainer" containerID="2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"
Oct 30 07:45:06.036848 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.036722    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl_odh-ai-gateway-infra(5cfab278-76ce-4cdb-869e-c1c7b192c490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490"
Oct 30 07:45:06.038325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.038268    6738 generic.go:356] "Generic (PLEG): container finished" podID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerID="f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511" exitCode=60
Oct 30 07:45:06.038325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.038317    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerDied","Data":"f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"}
Oct 30 07:45:06.038601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.038583    6738 scope.go:122] "RemoveContainer" containerID="f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"
Oct 30 07:45:06.038871 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.038846    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp_odh-ai-gateway-infra(15fdb9b6-4633-4f67-881e-33f3bab8d110)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110"
Oct 30 07:45:06.040236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.040215    6738 generic.go:356] "Generic (PLEG): container finished" podID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerID="20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5" exitCode=60
Oct 30 07:45:06.040352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.040338    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerDied","Data":"20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5"}
Oct 30 07:45:06.040491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.040477    6738 scope.go:122] "RemoveContainer" containerID="20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5"
Oct 30 07:45:06.040726 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.040705    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx_odh-ai-gateway-infra(83f358fa-509d-4e9f-bc0d-e0b5261a0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9"
Oct 30 07:45:06.042156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.042134    6738 generic.go:356] "Generic (PLEG): container finished" podID="26cab399-d2e1-4535-95a8-af6877be8c00" containerID="b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b" exitCode=60
Oct 30 07:45:06.042261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.042208    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerDied","Data":"b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b"}
Oct 30 07:45:06.042428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.042413    6738 scope.go:122] "RemoveContainer" containerID="b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b"
Oct 30 07:45:06.042689 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.042653    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889105-ggxfv_odh-ai-gateway-infra(26cab399-d2e1-4535-95a8-af6877be8c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" podUID="26cab399-d2e1-4535-95a8-af6877be8c00"
Oct 30 07:45:06.044061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.044040    6738 generic.go:356] "Generic (PLEG): container finished" podID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerID="c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d" exitCode=60
Oct 30 07:45:06.044149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.044119    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerDied","Data":"c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"}
Oct 30 07:45:06.044329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.044304    6738 scope.go:122] "RemoveContainer" containerID="c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"
Oct 30 07:45:06.044553 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.044533    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2_odh-ai-gateway-infra(da5d32a9-515b-415f-85a7-45d6e50d2c18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18"
Oct 30 07:45:06.045996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.045977    6738 generic.go:356] "Generic (PLEG): container finished" podID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerID="9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf" exitCode=60
Oct 30 07:45:06.046081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.046038    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerDied","Data":"9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf"}
Oct 30 07:45:06.046243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.046230    6738 scope.go:122] "RemoveContainer" containerID="9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf"
Oct 30 07:45:06.046495 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.046475    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29_odh-ai-gateway-infra(160ab89d-32a5-47bd-b0a6-7dbb5d83407b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b"
Oct 30 07:45:06.047785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.047768    6738 generic.go:356] "Generic (PLEG): container finished" podID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerID="8f28aa6fbe0b98b5b83ca41c07f262fd777365393175a84e6bcbfb0deee0383a" exitCode=60
Oct 30 07:45:06.047840 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.047828    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerDied","Data":"8f28aa6fbe0b98b5b83ca41c07f262fd777365393175a84e6bcbfb0deee0383a"}
Oct 30 07:45:06.048098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.048084    6738 scope.go:122] "RemoveContainer" containerID="8f28aa6fbe0b98b5b83ca41c07f262fd777365393175a84e6bcbfb0deee0383a"
Oct 30 07:45:06.048324 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:06.048287    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r_odh-ai-gateway-infra(ad73a2fe-05e8-4675-ab21-f422ea65b18f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f"
Oct 30 07:45:06.151656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.151630    6738 scope.go:122] "RemoveContainer" containerID="27c111cea6a80cd407b09ea2d0fb268d47b1a88a660187a6df8736e8069f5356"
Oct 30 07:45:06.167701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.167673    6738 scope.go:122] "RemoveContainer" containerID="83226db011a88e8d2dd692b495d055c91957c5d1c3604dfe855ea96aea8a55c2"
Oct 30 07:45:06.234920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.234896    6738 scope.go:122] "RemoveContainer" containerID="4c0d3c8503869ad6a067241c1013aecd3c9b66edbaa196f4556674e040d6a49e"
Oct 30 07:45:06.250459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.250435    6738 scope.go:122] "RemoveContainer" containerID="bbda1399f815fb1eb43d2806f39afa32295697de5b54657c8303feb8fd1fdb62"
Oct 30 07:45:06.265637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.265610    6738 scope.go:122] "RemoveContainer" containerID="2fdb9c320c04b0786f89f77010749581adb52cfb0dd2e09851baf5670ebff3a7"
Oct 30 07:45:06.337036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.337010    6738 scope.go:122] "RemoveContainer" containerID="e3553f15e57d2fbdf595794488fda96e4fd6c5c8e4799d01a2352875f0594a45"
Oct 30 07:45:06.354912 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.354890    6738 scope.go:122] "RemoveContainer" containerID="fdc5b7f8eab4b63ae8c4baf67c464cb820304d50209392c8cf6b7463427039a9"
Oct 30 07:45:06.370769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.370743    6738 scope.go:122] "RemoveContainer" containerID="0afef1b3b08518fe7c09f56ef36d77dc495090f70211d1e0a18b13b5204f842a"
Oct 30 07:45:06.386895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.386842    6738 scope.go:122] "RemoveContainer" containerID="7acd71c97f25974662a2599b5007eac0f54c1bfdda5af1542b1eafcc386f0ad0"
Oct 30 07:45:06.403800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.403773    6738 scope.go:122] "RemoveContainer" containerID="555ee9ee6e6ffeb6963a029b759ee47a2aa47a93170c04727107ef6fc64a70c5"
Oct 30 07:45:06.424692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:06.424660    6738 scope.go:122] "RemoveContainer" containerID="3ebe31ff019ae2bdb59a6c0cabc21a9355fee9932a793611188308103317cade"
Oct 30 07:45:07.059939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.059909    6738 scope.go:122] "RemoveContainer" containerID="15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d"
Oct 30 07:45:07.060560 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.060377    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw_odh-ai-gateway-infra(09e994a2-e628-419c-b0dd-58e677cdf9fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc"
Oct 30 07:45:07.061868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.061844    6738 scope.go:122] "RemoveContainer" containerID="14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"
Oct 30 07:45:07.062260 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.062238    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk_odh-ai-gateway-infra(5096ae27-38bc-4d4e-ad83-1efcaf982889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889"
Oct 30 07:45:07.063670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.063646    6738 scope.go:122] "RemoveContainer" containerID="2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"
Oct 30 07:45:07.064023 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.064002    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl_odh-ai-gateway-infra(5cfab278-76ce-4cdb-869e-c1c7b192c490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490"
Oct 30 07:45:07.065158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.065142    6738 scope.go:122] "RemoveContainer" containerID="f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"
Oct 30 07:45:07.065465 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.065444    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp_odh-ai-gateway-infra(15fdb9b6-4633-4f67-881e-33f3bab8d110)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110"
Oct 30 07:45:07.066872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.066854    6738 scope.go:122] "RemoveContainer" containerID="20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5"
Oct 30 07:45:07.067136 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.067116    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx_odh-ai-gateway-infra(83f358fa-509d-4e9f-bc0d-e0b5261a0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9"
Oct 30 07:45:07.068874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.068856    6738 scope.go:122] "RemoveContainer" containerID="b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b"
Oct 30 07:45:07.069118 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.069100    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889105-ggxfv_odh-ai-gateway-infra(26cab399-d2e1-4535-95a8-af6877be8c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" podUID="26cab399-d2e1-4535-95a8-af6877be8c00"
Oct 30 07:45:07.071028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.070881    6738 scope.go:122] "RemoveContainer" containerID="c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"
Oct 30 07:45:07.071200 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.071169    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2_odh-ai-gateway-infra(da5d32a9-515b-415f-85a7-45d6e50d2c18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18"
Oct 30 07:45:07.072916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.072892    6738 scope.go:122] "RemoveContainer" containerID="9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf"
Oct 30 07:45:07.073177 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.073156    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29_odh-ai-gateway-infra(160ab89d-32a5-47bd-b0a6-7dbb5d83407b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b"
Oct 30 07:45:07.076799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.076776    6738 generic.go:356] "Generic (PLEG): container finished" podID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerID="371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3" exitCode=60
Oct 30 07:45:07.076927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.076853    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerDied","Data":"371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"}
Oct 30 07:45:07.076927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.076914    6738 scope.go:122] "RemoveContainer" containerID="e9257ff8f1dbdcf4af54b74814a41dfef85709ab178d1305f4a90fd18935f675"
Oct 30 07:45:07.077131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.077112    6738 scope.go:122] "RemoveContainer" containerID="371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"
Oct 30 07:45:07.077483 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.077458    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8_odh-ai-gateway-infra(79e20dd7-eea4-4d0d-a447-6d4d8afed2ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed"
Oct 30 07:45:07.079228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.079205    6738 generic.go:356] "Generic (PLEG): container finished" podID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerID="d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309" exitCode=60
Oct 30 07:45:07.079385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.079282    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerDied","Data":"d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"}
Oct 30 07:45:07.079644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.079625    6738 scope.go:122] "RemoveContainer" containerID="d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"
Oct 30 07:45:07.079973 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.079944    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8_odh-ai-gateway-infra(6222c938-39d6-4d25-b97a-22be3e24e33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b"
Oct 30 07:45:07.081652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.081631    6738 scope.go:122] "RemoveContainer" containerID="ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"
Oct 30 07:45:07.081984 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.081964    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8_odh-ai-gateway-infra(3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c"
Oct 30 07:45:07.083516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.083497    6738 scope.go:122] "RemoveContainer" containerID="7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff"
Oct 30 07:45:07.083796 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.083774    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k_odh-ai-gateway-infra(7256a39a-061a-418e-b299-e67eacf4f780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" podUID="7256a39a-061a-418e-b299-e67eacf4f780"
Oct 30 07:45:07.085268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.085250    6738 scope.go:122] "RemoveContainer" containerID="4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf"
Oct 30 07:45:07.085537 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.085518    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9_odh-ai-gateway-infra(f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f"
Oct 30 07:45:07.094142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.094121    6738 scope.go:122] "RemoveContainer" containerID="c814ee97b8c14865ef40427187cb834d9c7cfc5ae75477633db836f9774f4f26"
Oct 30 07:45:07.381883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:07.381840    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:07.522660 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:07.522622    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:45:08.098269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:08.098240    6738 scope.go:122] "RemoveContainer" containerID="371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"
Oct 30 07:45:08.098802 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:08.098724    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8_odh-ai-gateway-infra(79e20dd7-eea4-4d0d-a447-6d4d8afed2ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed"
Oct 30 07:45:08.100329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:08.100288    6738 scope.go:122] "RemoveContainer" containerID="d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"
Oct 30 07:45:08.100697 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:08.100672    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8_odh-ai-gateway-infra(6222c938-39d6-4d25-b97a-22be3e24e33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b"
Oct 30 07:45:09.209366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:09.209288    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:11.033443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:11.033404    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"]
Oct 30 07:45:11.045076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:11.045041    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0596e6fa-ggjft"]
Oct 30 07:45:11.060111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:11.060067    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:11.693668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:11.693629    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f892d204-36e4-4400-adb2-8a4befcc6754" path="/var/lib/kubelet/pods/f892d204-36e4-4400-adb2-8a4befcc6754/volumes"
Oct 30 07:45:12.774705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:12.774671    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:12.900659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:12.900622    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:14.913093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:14.913050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:15.026138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:15.026094    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:15.683355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:15.683321    6738 scope.go:122] "RemoveContainer" containerID="b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3"
Oct 30 07:45:15.683834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:15.683801    6738 scope.go:122] "RemoveContainer" containerID="2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"
Oct 30 07:45:16.205090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:16.205058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerStarted","Data":"d4f661f78482622eecafee26fcf4a77b081e33e4b291181c51a5fa341400b17d"}
Oct 30 07:45:16.683049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:16.683018    6738 scope.go:122] "RemoveContainer" containerID="14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"
Oct 30 07:45:16.789021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:16.788982    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:17.216762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.216714    6738 generic.go:356] "Generic (PLEG): container finished" podID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerID="6db0216ed96cb33449b9a6ef5bdafcb7f05d3f5525e5a7bd89503eac38f48ac1" exitCode=60
Oct 30 07:45:17.217194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.216785    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerDied","Data":"6db0216ed96cb33449b9a6ef5bdafcb7f05d3f5525e5a7bd89503eac38f48ac1"}
Oct 30 07:45:17.217194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.216844    6738 scope.go:122] "RemoveContainer" containerID="b6111bd30e4fe08457784275002c3d93d340159834958b3e77e054a9c8d973e3"
Oct 30 07:45:17.217328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.217285    6738 scope.go:122] "RemoveContainer" containerID="6db0216ed96cb33449b9a6ef5bdafcb7f05d3f5525e5a7bd89503eac38f48ac1"
Oct 30 07:45:17.217719 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:17.217696    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl_odh-ai-gateway-infra(3b12e79b-7d3f-4ee9-96c6-4d6d8533d299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299"
Oct 30 07:45:17.219337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.219317    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerStarted","Data":"cdd52ddda1a8c6da726080de3d6659774312e67c025d63e67f868774732e5269"}
Oct 30 07:45:17.221371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.221350    6738 generic.go:356] "Generic (PLEG): container finished" podID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerID="d4f661f78482622eecafee26fcf4a77b081e33e4b291181c51a5fa341400b17d" exitCode=60
Oct 30 07:45:17.221474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.221423    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerDied","Data":"d4f661f78482622eecafee26fcf4a77b081e33e4b291181c51a5fa341400b17d"}
Oct 30 07:45:17.221786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.221772    6738 scope.go:122] "RemoveContainer" containerID="d4f661f78482622eecafee26fcf4a77b081e33e4b291181c51a5fa341400b17d"
Oct 30 07:45:17.222050 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:17.222034    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl_odh-ai-gateway-infra(5cfab278-76ce-4cdb-869e-c1c7b192c490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490"
Oct 30 07:45:17.232014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.231962    6738 scope.go:122] "RemoveContainer" containerID="2f089f29d32bc5e93623309cf20895870f92aba8cb083bd492e3e4ecd6e876c5"
Oct 30 07:45:17.258361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.258326    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:17.683590 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.683554    6738 scope.go:122] "RemoveContainer" containerID="8f28aa6fbe0b98b5b83ca41c07f262fd777365393175a84e6bcbfb0deee0383a"
Oct 30 07:45:17.683770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:17.683710    6738 scope.go:122] "RemoveContainer" containerID="c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"
Oct 30 07:45:18.240083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.239993    6738 generic.go:356] "Generic (PLEG): container finished" podID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerID="cdd52ddda1a8c6da726080de3d6659774312e67c025d63e67f868774732e5269" exitCode=60
Oct 30 07:45:18.240083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.240070    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerDied","Data":"cdd52ddda1a8c6da726080de3d6659774312e67c025d63e67f868774732e5269"}
Oct 30 07:45:18.240649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.240161    6738 scope.go:122] "RemoveContainer" containerID="14809ffd5074ba2aa88217dbe861fc26fb909b38429e142c7cb472fbbc54ff5f"
Oct 30 07:45:18.240695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.240661    6738 scope.go:122] "RemoveContainer" containerID="cdd52ddda1a8c6da726080de3d6659774312e67c025d63e67f868774732e5269"
Oct 30 07:45:18.241088 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:18.241060    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk_odh-ai-gateway-infra(5096ae27-38bc-4d4e-ad83-1efcaf982889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889"
Oct 30 07:45:18.244331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.244305    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerStarted","Data":"bf9331abdecb5579108549ea7e4969cb24bd228e408c387869fd7b2a0f7e4ac9"}
Oct 30 07:45:18.246658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.246637    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerStarted","Data":"e84d54829012c80acee6b693d5a1b3be751cff26c287987ad363ab40cce81a77"}
Oct 30 07:45:18.275229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.275192    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:18.407379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.407340    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:18.499230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.499190    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:18.544088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.544060    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:18.562658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.562622    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\" (UniqueName: \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\") pod \"5cfab278-76ce-4cdb-869e-c1c7b192c490\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") "
Oct 30 07:45:18.562955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.562938    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\") pod \"5cfab278-76ce-4cdb-869e-c1c7b192c490\" (UID: \"5cfab278-76ce-4cdb-869e-c1c7b192c490\") "
Oct 30 07:45:18.563389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.563361    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca" pod "5cfab278-76ce-4cdb-869e-c1c7b192c490" (UID: "5cfab278-76ce-4cdb-869e-c1c7b192c490"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:18.563715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.563696    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfab278-76ce-4cdb-869e-c1c7b192c490-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.565845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.565814    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z" pod "5cfab278-76ce-4cdb-869e-c1c7b192c490" (UID: "5cfab278-76ce-4cdb-869e-c1c7b192c490"). InnerVolumeSpecName "kube-api-access-wks2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:18.665671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.665628    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\") pod \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") "
Oct 30 07:45:18.665920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.665900    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\" (UniqueName: \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\") pod \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\" (UID: \"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299\") "
Oct 30 07:45:18.666607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.666579    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wks2z\" (UniqueName: \"kubernetes.io/projected/5cfab278-76ce-4cdb-869e-c1c7b192c490-kube-api-access-wks2z\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.667383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.667353    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca" pod "3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" (UID: "3b12e79b-7d3f-4ee9-96c6-4d6d8533d299"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:18.669589 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.669561    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l" pod "3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" (UID: "3b12e79b-7d3f-4ee9-96c6-4d6d8533d299"). InnerVolumeSpecName "kube-api-access-66g9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:18.683524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.683493    6738 scope.go:122] "RemoveContainer" containerID="15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d"
Oct 30 07:45:18.683940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.683914    6738 scope.go:122] "RemoveContainer" containerID="9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf"
Oct 30 07:45:18.684103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.684076    6738 scope.go:122] "RemoveContainer" containerID="371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"
Oct 30 07:45:18.684227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.684207    6738 scope.go:122] "RemoveContainer" containerID="20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5"
Oct 30 07:45:18.684429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.684390    6738 scope.go:122] "RemoveContainer" containerID="ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"
Oct 30 07:45:18.755821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.755792    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:18.772287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.772134    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.772287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.772159    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66g9l\" (UniqueName: \"kubernetes.io/projected/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299-kube-api-access-66g9l\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.895826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:18.895794    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:19.258230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.258192    6738 generic.go:356] "Generic (PLEG): container finished" podID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerID="e84d54829012c80acee6b693d5a1b3be751cff26c287987ad363ab40cce81a77" exitCode=60
Oct 30 07:45:19.258751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.258275    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerDied","Data":"e84d54829012c80acee6b693d5a1b3be751cff26c287987ad363ab40cce81a77"}
Oct 30 07:45:19.258751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.258359    6738 scope.go:122] "RemoveContainer" containerID="8f28aa6fbe0b98b5b83ca41c07f262fd777365393175a84e6bcbfb0deee0383a"
Oct 30 07:45:19.260221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.260199    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"
Oct 30 07:45:19.260509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.260222    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl" event={"ID":"3b12e79b-7d3f-4ee9-96c6-4d6d8533d299","Type":"ContainerDied","Data":"642ab35dbc97c72849a1f33b55ef01715176bc30806d13f70820cf39b57412bd"}
Oct 30 07:45:19.262814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.262778    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerStarted","Data":"38b16a5db4a3193414fb5ecf289b323cd63afd69f9c9ab1d8d036283bae50b9e"}
Oct 30 07:45:19.266682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.266577    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"
Oct 30 07:45:19.266682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.266623    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl" event={"ID":"5cfab278-76ce-4cdb-869e-c1c7b192c490","Type":"ContainerDied","Data":"abc17da08a83da5e7dc94c29af8772ef61caf97b927d06e8468e845e7e0abc38"}
Oct 30 07:45:19.273149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.271131    6738 generic.go:356] "Generic (PLEG): container finished" podID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerID="bf9331abdecb5579108549ea7e4969cb24bd228e408c387869fd7b2a0f7e4ac9" exitCode=60
Oct 30 07:45:19.273149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.271194    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerDied","Data":"bf9331abdecb5579108549ea7e4969cb24bd228e408c387869fd7b2a0f7e4ac9"}
Oct 30 07:45:19.273149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.271918    6738 scope.go:122] "RemoveContainer" containerID="bf9331abdecb5579108549ea7e4969cb24bd228e408c387869fd7b2a0f7e4ac9"
Oct 30 07:45:19.273149 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:19.272384    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2_odh-ai-gateway-infra(da5d32a9-515b-415f-85a7-45d6e50d2c18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18"
Oct 30 07:45:19.279401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.279365    6738 scope.go:122] "RemoveContainer" containerID="6db0216ed96cb33449b9a6ef5bdafcb7f05d3f5525e5a7bd89503eac38f48ac1"
Oct 30 07:45:19.295281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.295247    6738 scope.go:122] "RemoveContainer" containerID="d4f661f78482622eecafee26fcf4a77b081e33e4b291181c51a5fa341400b17d"
Oct 30 07:45:19.321452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.321414    6738 scope.go:122] "RemoveContainer" containerID="c9f2d4156ea14720cd39aa0ffb7efbf4d833764ebe81b6dc7408656f3939055d"
Oct 30 07:45:19.424435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.424385    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:19.430217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.430180    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:19.435790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.435757    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889105-fb4fl"]
Oct 30 07:45:19.516415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.516375    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:19.530630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.529470    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889105-wljfl"]
Oct 30 07:45:19.535660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.535634    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:19.581736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.581595    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:19.705550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.705512    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\" (UniqueName: \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\") pod \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") "
Oct 30 07:45:19.705777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.705637    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\" (UniqueName: \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\") pod \"5096ae27-38bc-4d4e-ad83-1efcaf982889\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") "
Oct 30 07:45:19.705777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.705663    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\") pod \"5096ae27-38bc-4d4e-ad83-1efcaf982889\" (UID: \"5096ae27-38bc-4d4e-ad83-1efcaf982889\") "
Oct 30 07:45:19.705777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.705707    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\") pod \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\" (UID: \"ad73a2fe-05e8-4675-ab21-f422ea65b18f\") "
Oct 30 07:45:19.706655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.706633    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca" pod "ad73a2fe-05e8-4675-ab21-f422ea65b18f" (UID: "ad73a2fe-05e8-4675-ab21-f422ea65b18f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:19.707236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.707206    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" path="/var/lib/kubelet/pods/3b12e79b-7d3f-4ee9-96c6-4d6d8533d299/volumes"
Oct 30 07:45:19.707664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.707646    6738 scope.go:122] "RemoveContainer" containerID="d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"
Oct 30 07:45:19.707849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.707831    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" path="/var/lib/kubelet/pods/5cfab278-76ce-4cdb-869e-c1c7b192c490/volumes"
Oct 30 07:45:19.708240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.708219    6738 scope.go:122] "RemoveContainer" containerID="4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf"
Oct 30 07:45:19.708882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.708858    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x" pod "ad73a2fe-05e8-4675-ab21-f422ea65b18f" (UID: "ad73a2fe-05e8-4675-ab21-f422ea65b18f"). InnerVolumeSpecName "kube-api-access-9r72x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:19.709813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.709759    6738 scope.go:122] "RemoveContainer" containerID="7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff"
Oct 30 07:45:19.710377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.710352    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca" pod "5096ae27-38bc-4d4e-ad83-1efcaf982889" (UID: "5096ae27-38bc-4d4e-ad83-1efcaf982889"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:19.710915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.710872    6738 scope.go:122] "RemoveContainer" containerID="b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b"
Oct 30 07:45:19.711358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.711282    6738 scope.go:122] "RemoveContainer" containerID="f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"
Oct 30 07:45:19.711473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.711453    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7" pod "5096ae27-38bc-4d4e-ad83-1efcaf982889" (UID: "5096ae27-38bc-4d4e-ad83-1efcaf982889"). InnerVolumeSpecName "kube-api-access-r7qc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:19.808755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.808713    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r72x\" (UniqueName: \"kubernetes.io/projected/ad73a2fe-05e8-4675-ab21-f422ea65b18f-kube-api-access-9r72x\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.808929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.808911    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qc7\" (UniqueName: \"kubernetes.io/projected/5096ae27-38bc-4d4e-ad83-1efcaf982889-kube-api-access-r7qc7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.809031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.809018    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096ae27-38bc-4d4e-ad83-1efcaf982889-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.809123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:19.809111    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73a2fe-05e8-4675-ab21-f422ea65b18f-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.282550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.282511    6738 generic.go:356] "Generic (PLEG): container finished" podID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerID="2701bba2be24fafd8c241e689660ee9fa7ddf60b1da8039dbc91596e146ea4ba" exitCode=60
Oct 30 07:45:20.283017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.282585    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerDied","Data":"2701bba2be24fafd8c241e689660ee9fa7ddf60b1da8039dbc91596e146ea4ba"}
Oct 30 07:45:20.283017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.282692    6738 scope.go:122] "RemoveContainer" containerID="371ff7dde0ac7f509c40af6fc6e951870e98a47c46c1835385ef42f886ff64c3"
Oct 30 07:45:20.283172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.283155    6738 scope.go:122] "RemoveContainer" containerID="2701bba2be24fafd8c241e689660ee9fa7ddf60b1da8039dbc91596e146ea4ba"
Oct 30 07:45:20.283625 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:20.283601    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8_odh-ai-gateway-infra(79e20dd7-eea4-4d0d-a447-6d4d8afed2ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed"
Oct 30 07:45:20.286399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.286375    6738 generic.go:356] "Generic (PLEG): container finished" podID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerID="fe3c299a6c5d141b80822018b37f5ae7bd36f4f3a58c0d72b697100afbe2b472" exitCode=60
Oct 30 07:45:20.286500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.286446    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerDied","Data":"fe3c299a6c5d141b80822018b37f5ae7bd36f4f3a58c0d72b697100afbe2b472"}
Oct 30 07:45:20.286906 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.286890    6738 scope.go:122] "RemoveContainer" containerID="fe3c299a6c5d141b80822018b37f5ae7bd36f4f3a58c0d72b697100afbe2b472"
Oct 30 07:45:20.287395 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:20.287373    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8_odh-ai-gateway-infra(3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c"
Oct 30 07:45:20.288556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.288527    6738 generic.go:356] "Generic (PLEG): container finished" podID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerID="38b16a5db4a3193414fb5ecf289b323cd63afd69f9c9ab1d8d036283bae50b9e" exitCode=60
Oct 30 07:45:20.288671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.288598    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerDied","Data":"38b16a5db4a3193414fb5ecf289b323cd63afd69f9c9ab1d8d036283bae50b9e"}
Oct 30 07:45:20.289044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.289015    6738 scope.go:122] "RemoveContainer" containerID="38b16a5db4a3193414fb5ecf289b323cd63afd69f9c9ab1d8d036283bae50b9e"
Oct 30 07:45:20.289454 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:20.289431    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw_odh-ai-gateway-infra(09e994a2-e628-419c-b0dd-58e677cdf9fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc"
Oct 30 07:45:20.290361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.290334    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"
Oct 30 07:45:20.290455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.290336    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk" event={"ID":"5096ae27-38bc-4d4e-ad83-1efcaf982889","Type":"ContainerDied","Data":"893923bd1074a9d189ce8bee12a0ef02a2b319d0a039db00b1b11059814a88e4"}
Oct 30 07:45:20.293879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.293858    6738 generic.go:356] "Generic (PLEG): container finished" podID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerID="7909dc0334ea4847c9b5e66f0edfc3461ba751d4abdcf41c5d2088fb90e31f68" exitCode=60
Oct 30 07:45:20.293975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.293925    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerDied","Data":"7909dc0334ea4847c9b5e66f0edfc3461ba751d4abdcf41c5d2088fb90e31f68"}
Oct 30 07:45:20.294368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.294351    6738 scope.go:122] "RemoveContainer" containerID="7909dc0334ea4847c9b5e66f0edfc3461ba751d4abdcf41c5d2088fb90e31f68"
Oct 30 07:45:20.294733 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:20.294711    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx_odh-ai-gateway-infra(83f358fa-509d-4e9f-bc0d-e0b5261a0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9"
Oct 30 07:45:20.297869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.297838    6738 generic.go:356] "Generic (PLEG): container finished" podID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerID="e42d1b8f56b35fc54fc56ea42a818d8cc598e87d63016bba5e532fdb22067be4" exitCode=60
Oct 30 07:45:20.297978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.297903    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerDied","Data":"e42d1b8f56b35fc54fc56ea42a818d8cc598e87d63016bba5e532fdb22067be4"}
Oct 30 07:45:20.298530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.298504    6738 scope.go:122] "RemoveContainer" containerID="e42d1b8f56b35fc54fc56ea42a818d8cc598e87d63016bba5e532fdb22067be4"
Oct 30 07:45:20.298931 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:20.298908    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29_odh-ai-gateway-infra(160ab89d-32a5-47bd-b0a6-7dbb5d83407b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b"
Oct 30 07:45:20.300072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.300051    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"
Oct 30 07:45:20.300072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.300060    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r" event={"ID":"ad73a2fe-05e8-4675-ab21-f422ea65b18f","Type":"ContainerDied","Data":"14611554f49d7bff37afc2f5b6296bf0dcaf2d6d8bf3379323848ebd04449b75"}
Oct 30 07:45:20.302733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.302713    6738 scope.go:122] "RemoveContainer" containerID="ec3f4bc3237f8d87f47e9c9e76aeccc234a297194dc1c5234e792f11b2e6bf49"
Oct 30 07:45:20.319719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.319692    6738 scope.go:122] "RemoveContainer" containerID="15f6cf6b96dc1f9b3484ba397d789b1502713e2ea7b118d9897657ddd476f15d"
Oct 30 07:45:20.339582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.339356    6738 scope.go:122] "RemoveContainer" containerID="cdd52ddda1a8c6da726080de3d6659774312e67c025d63e67f868774732e5269"
Oct 30 07:45:20.352887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.352866    6738 scope.go:122] "RemoveContainer" containerID="20fe474357f710259f6f11a923b1948af7b9b1afc1c9e3fc647b0dc96d25c6f5"
Oct 30 07:45:20.375375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.375331    6738 scope.go:122] "RemoveContainer" containerID="9eacda7e58b3d6f532ff9f51065bd02695aa03ac9448a531d5243585f1f6cfcf"
Oct 30 07:45:20.418453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.418394    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:20.425400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.425365    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:20.430465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.430438    6738 scope.go:122] "RemoveContainer" containerID="e84d54829012c80acee6b693d5a1b3be751cff26c287987ad363ab40cce81a77"
Oct 30 07:45:20.432176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.432080    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889105-j7ztk"]
Oct 30 07:45:20.535384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.535352    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:20.701686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.701648    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:20.708016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.707988    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8f1df-29889105-j7v6r"]
Oct 30 07:45:20.726868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.726836    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\") pod \"da5d32a9-515b-415f-85a7-45d6e50d2c18\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") "
Oct 30 07:45:20.727216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.727160    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\" (UniqueName: \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\") pod \"da5d32a9-515b-415f-85a7-45d6e50d2c18\" (UID: \"da5d32a9-515b-415f-85a7-45d6e50d2c18\") "
Oct 30 07:45:20.727270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.727251    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca" pod "da5d32a9-515b-415f-85a7-45d6e50d2c18" (UID: "da5d32a9-515b-415f-85a7-45d6e50d2c18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:20.727975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.727921    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d32a9-515b-415f-85a7-45d6e50d2c18-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.729494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.729471    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p" pod "da5d32a9-515b-415f-85a7-45d6e50d2c18" (UID: "da5d32a9-515b-415f-85a7-45d6e50d2c18"). InnerVolumeSpecName "kube-api-access-fx86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:20.829432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.829396    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx86p\" (UniqueName: \"kubernetes.io/projected/da5d32a9-515b-415f-85a7-45d6e50d2c18-kube-api-access-fx86p\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.962723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:20.962689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:21.315534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.315500    6738 generic.go:356] "Generic (PLEG): container finished" podID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerID="73c05da480396918a14190cebada7dce891ccb2f513ce40a92b8bb5d53ee5a5b" exitCode=60
Oct 30 07:45:21.316019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.315538    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerDied","Data":"73c05da480396918a14190cebada7dce891ccb2f513ce40a92b8bb5d53ee5a5b"}
Oct 30 07:45:21.316019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.315597    6738 scope.go:122] "RemoveContainer" containerID="d7c32ccbc79eb21f00fbdc5ba2a229060882663ccedb3809d99b8045b16c6309"
Oct 30 07:45:21.316117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.316100    6738 scope.go:122] "RemoveContainer" containerID="73c05da480396918a14190cebada7dce891ccb2f513ce40a92b8bb5d53ee5a5b"
Oct 30 07:45:21.316815 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:21.316558    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8_odh-ai-gateway-infra(6222c938-39d6-4d25-b97a-22be3e24e33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b"
Oct 30 07:45:21.319592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.319570    6738 generic.go:356] "Generic (PLEG): container finished" podID="7256a39a-061a-418e-b299-e67eacf4f780" containerID="2758490c805a55803b100c0cdfaaf17699c4ce1de9dc673bee77b5a704982656" exitCode=60
Oct 30 07:45:21.319710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.319595    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerDied","Data":"2758490c805a55803b100c0cdfaaf17699c4ce1de9dc673bee77b5a704982656"}
Oct 30 07:45:21.320057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.320041    6738 scope.go:122] "RemoveContainer" containerID="2758490c805a55803b100c0cdfaaf17699c4ce1de9dc673bee77b5a704982656"
Oct 30 07:45:21.320520 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:21.320496    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k_odh-ai-gateway-infra(7256a39a-061a-418e-b299-e67eacf4f780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" podUID="7256a39a-061a-418e-b299-e67eacf4f780"
Oct 30 07:45:21.321765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.321744    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerID="f61137e10cb8a1f190b55a09f5b55d132a04473f0f424cd877ce29dcab3702cd" exitCode=60
Oct 30 07:45:21.321863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.321818    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerDied","Data":"f61137e10cb8a1f190b55a09f5b55d132a04473f0f424cd877ce29dcab3702cd"}
Oct 30 07:45:21.322289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.322273    6738 scope.go:122] "RemoveContainer" containerID="f61137e10cb8a1f190b55a09f5b55d132a04473f0f424cd877ce29dcab3702cd"
Oct 30 07:45:21.322696 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:21.322672    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9_odh-ai-gateway-infra(f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f"
Oct 30 07:45:21.327221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.327190    6738 generic.go:356] "Generic (PLEG): container finished" podID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerID="1fdccaf5c901b7be8ef2a7f579f6969cc679ac2bba079cc204656f1daa8d9071" exitCode=60
Oct 30 07:45:21.327341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.327229    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerDied","Data":"1fdccaf5c901b7be8ef2a7f579f6969cc679ac2bba079cc204656f1daa8d9071"}
Oct 30 07:45:21.327707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.327684    6738 scope.go:122] "RemoveContainer" containerID="1fdccaf5c901b7be8ef2a7f579f6969cc679ac2bba079cc204656f1daa8d9071"
Oct 30 07:45:21.328095 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:21.328074    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp_odh-ai-gateway-infra(15fdb9b6-4633-4f67-881e-33f3bab8d110)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110"
Oct 30 07:45:21.331286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.331265    6738 generic.go:356] "Generic (PLEG): container finished" podID="26cab399-d2e1-4535-95a8-af6877be8c00" containerID="0be993312d62fb940e73bc8545c81b6bdb905b77e98e264cd9faf5a3185ff884" exitCode=60
Oct 30 07:45:21.331401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.331325    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerDied","Data":"0be993312d62fb940e73bc8545c81b6bdb905b77e98e264cd9faf5a3185ff884"}
Oct 30 07:45:21.331824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.331800    6738 scope.go:122] "RemoveContainer" containerID="0be993312d62fb940e73bc8545c81b6bdb905b77e98e264cd9faf5a3185ff884"
Oct 30 07:45:21.332127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.332106    6738 scope.go:122] "RemoveContainer" containerID="7a69a7717eb5c3c56eda6dae89ea6f700afa82606f7fe13cef77fe9ad8e03bff"
Oct 30 07:45:21.332227 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:21.332200    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889105-ggxfv_odh-ai-gateway-infra(26cab399-d2e1-4535-95a8-af6877be8c00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" podUID="26cab399-d2e1-4535-95a8-af6877be8c00"
Oct 30 07:45:21.332939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.332907    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"
Oct 30 07:45:21.333036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.332949    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2" event={"ID":"da5d32a9-515b-415f-85a7-45d6e50d2c18","Type":"ContainerDied","Data":"9fd7a9f2379e43027da48ba78647d497fa3b47054b8f9fda891b5fa610c3fe41"}
Oct 30 07:45:21.350235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.350210    6738 scope.go:122] "RemoveContainer" containerID="4217e8cc9955daf9faa405615e6e327a76e5f7d8a363a57bc897e5fa2eeb6fdf"
Oct 30 07:45:21.367280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.367248    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:21.380008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.379969    6738 scope.go:122] "RemoveContainer" containerID="f339230d906ef9b706b7dea2d37a7f503a151979fd8f48c7558ab6aa1add8511"
Oct 30 07:45:21.399504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.399476    6738 scope.go:122] "RemoveContainer" containerID="b5337982a6271657cca867dfa1418f331d984c37b60c6196cf131e1476fa6b6b"
Oct 30 07:45:21.450811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.450772    6738 scope.go:122] "RemoveContainer" containerID="bf9331abdecb5579108549ea7e4969cb24bd228e408c387869fd7b2a0f7e4ac9"
Oct 30 07:45:21.479983 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.479950    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:21.529656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.529618    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:21.563952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.563886    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:21.651219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.650730    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\" (UniqueName: \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\") pod \"09e994a2-e628-419c-b0dd-58e677cdf9fc\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") "
Oct 30 07:45:21.653430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.653041    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\") pod \"09e994a2-e628-419c-b0dd-58e677cdf9fc\" (UID: \"09e994a2-e628-419c-b0dd-58e677cdf9fc\") "
Oct 30 07:45:21.654187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.653485    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca" pod "09e994a2-e628-419c-b0dd-58e677cdf9fc" (UID: "09e994a2-e628-419c-b0dd-58e677cdf9fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:21.655471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.655365    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:21.656285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.655905    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e994a2-e628-419c-b0dd-58e677cdf9fc-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.667279 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.667232    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889105-m9wn2"]
Oct 30 07:45:21.676401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.676368    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm" pod "09e994a2-e628-419c-b0dd-58e677cdf9fc" (UID: "09e994a2-e628-419c-b0dd-58e677cdf9fc"). InnerVolumeSpecName "kube-api-access-gw8zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:21.678910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.678878    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:21.752157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.752122    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" path="/var/lib/kubelet/pods/5096ae27-38bc-4d4e-ad83-1efcaf982889/volumes"
Oct 30 07:45:21.753100 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.753063    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" path="/var/lib/kubelet/pods/ad73a2fe-05e8-4675-ab21-f422ea65b18f/volumes"
Oct 30 07:45:21.753879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.753858    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" path="/var/lib/kubelet/pods/da5d32a9-515b-415f-85a7-45d6e50d2c18/volumes"
Oct 30 07:45:21.758083 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.758062    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw8zm\" (UniqueName: \"kubernetes.io/projected/09e994a2-e628-419c-b0dd-58e677cdf9fc-kube-api-access-gw8zm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.759272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.759168    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:21.762438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.762416    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:21.804886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.804827    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:21.859694 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.859661    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\" (UniqueName: \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\") pod \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") "
Oct 30 07:45:21.859883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.859841    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\" (UniqueName: \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\") pod \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") "
Oct 30 07:45:21.860003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.859987    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\") pod \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") "
Oct 30 07:45:21.860085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.860020    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\" (UniqueName: \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\") pod \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\" (UID: \"160ab89d-32a5-47bd-b0a6-7dbb5d83407b\") "
Oct 30 07:45:21.860150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.860123    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\") pod \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\" (UID: \"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed\") "
Oct 30 07:45:21.860274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.860166    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\") pod \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\" (UID: \"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c\") "
Oct 30 07:45:21.861217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.861189    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca" pod "3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" (UID: "3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:21.865089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.864982    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca" pod "160ab89d-32a5-47bd-b0a6-7dbb5d83407b" (UID: "160ab89d-32a5-47bd-b0a6-7dbb5d83407b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:21.865577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.865255    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg" pod "79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" (UID: "79e20dd7-eea4-4d0d-a447-6d4d8afed2ed"). InnerVolumeSpecName "kube-api-access-r78qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:21.865863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.865830    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca" pod "79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" (UID: "79e20dd7-eea4-4d0d-a447-6d4d8afed2ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:21.865941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.865883    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl" pod "3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" (UID: "3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c"). InnerVolumeSpecName "kube-api-access-q9zwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:21.867338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.867306    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9" pod "160ab89d-32a5-47bd-b0a6-7dbb5d83407b" (UID: "160ab89d-32a5-47bd-b0a6-7dbb5d83407b"). InnerVolumeSpecName "kube-api-access-97lx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:21.898332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.897617    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:21.961466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.961424    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\" (UniqueName: \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\") pod \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") "
Oct 30 07:45:21.961786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.961762    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\") pod \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\" (UID: \"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9\") "
Oct 30 07:45:21.962182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962152    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca" pod "83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" (UID: "83f358fa-509d-4e9f-bc0d-e0b5261a0ee9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:21.962547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962521    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78qg\" (UniqueName: \"kubernetes.io/projected/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-kube-api-access-r78qg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962552    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962566    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lx9\" (UniqueName: \"kubernetes.io/projected/160ab89d-32a5-47bd-b0a6-7dbb5d83407b-kube-api-access-97lx9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962580    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962593    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962607    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9zwl\" (UniqueName: \"kubernetes.io/projected/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c-kube-api-access-q9zwl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.962652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.962620    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.963924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:21.963898    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g" pod "83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" (UID: "83f358fa-509d-4e9f-bc0d-e0b5261a0ee9"). InnerVolumeSpecName "kube-api-access-v427g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:22.064595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.064507    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v427g\" (UniqueName: \"kubernetes.io/projected/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9-kube-api-access-v427g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:22.351866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.351777    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"
Oct 30 07:45:22.351866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.351796    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8" event={"ID":"79e20dd7-eea4-4d0d-a447-6d4d8afed2ed","Type":"ContainerDied","Data":"819131edf8c4e0961106f8c81aef76be4135a5ec740b520704853c25f20a70f4"}
Oct 30 07:45:22.352461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.351916    6738 scope.go:122] "RemoveContainer" containerID="2701bba2be24fafd8c241e689660ee9fa7ddf60b1da8039dbc91596e146ea4ba"
Oct 30 07:45:22.355557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.355530    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"
Oct 30 07:45:22.355720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.355566    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8" event={"ID":"3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c","Type":"ContainerDied","Data":"b1b658703fc33e15e45a4dd22e608f7f64c6b724f1b2917107f13a9ffe36a36f"}
Oct 30 07:45:22.361278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.361255    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"
Oct 30 07:45:22.361278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.361268    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw" event={"ID":"09e994a2-e628-419c-b0dd-58e677cdf9fc","Type":"ContainerDied","Data":"0e4ce98970d00c87a3c0f3c532a00710b4f15e668bd662cefaaa3a9bba9c6989"}
Oct 30 07:45:22.365425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.365397    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"
Oct 30 07:45:22.365537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.365443    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx" event={"ID":"83f358fa-509d-4e9f-bc0d-e0b5261a0ee9","Type":"ContainerDied","Data":"e28f91115375ab196a137ffe836197683c2b1417ba8e2e76d8198945a752bf13"}
Oct 30 07:45:22.367341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.367306    6738 scope.go:122] "RemoveContainer" containerID="fe3c299a6c5d141b80822018b37f5ae7bd36f4f3a58c0d72b697100afbe2b472"
Oct 30 07:45:22.369833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.369811    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"
Oct 30 07:45:22.369915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.369810    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29" event={"ID":"160ab89d-32a5-47bd-b0a6-7dbb5d83407b","Type":"ContainerDied","Data":"4fb93ab602a4556c65b8e620dd7c9a168339bb3f2b7190f6fd1d307f6e7d417b"}
Oct 30 07:45:22.381620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.381080    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:22.382773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.382744    6738 scope.go:122] "RemoveContainer" containerID="38b16a5db4a3193414fb5ecf289b323cd63afd69f9c9ab1d8d036283bae50b9e"
Oct 30 07:45:22.398534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.398503    6738 scope.go:122] "RemoveContainer" containerID="7909dc0334ea4847c9b5e66f0edfc3461ba751d4abdcf41c5d2088fb90e31f68"
Oct 30 07:45:22.412839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.412815    6738 scope.go:122] "RemoveContainer" containerID="e42d1b8f56b35fc54fc56ea42a818d8cc598e87d63016bba5e532fdb22067be4"
Oct 30 07:45:22.426957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.426928    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:22.433105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.433073    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:22.438090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.438057    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889105-b8bw8"]
Oct 30 07:45:22.511809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.511475    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:22.522245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.519435    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889105-4zpj8"]
Oct 30 07:45:22.538473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.537175    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:22.559216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.559177    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:22.584362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.573373    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:22.584362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.580483    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889105-vwr29"]
Oct 30 07:45:22.644757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.638603    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:22.653550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.653505    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.682530    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\" (UniqueName: \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\") pod \"6222c938-39d6-4d25-b97a-22be3e24e33b\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") "
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.682775    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\") pod \"6222c938-39d6-4d25-b97a-22be3e24e33b\" (UID: \"6222c938-39d6-4d25-b97a-22be3e24e33b\") "
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.683940    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca" pod "6222c938-39d6-4d25-b97a-22be3e24e33b" (UID: "6222c938-39d6-4d25-b97a-22be3e24e33b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.683985    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.692330    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889105-4g7vw"]
Oct 30 07:45:22.697453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.692729    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr" pod "6222c938-39d6-4d25-b97a-22be3e24e33b" (UID: "6222c938-39d6-4d25-b97a-22be3e24e33b"). InnerVolumeSpecName "kube-api-access-f2pfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:22.746475 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.746404    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:22.782045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.781644    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:22.805392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.785153    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\" (UniqueName: \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\") pod \"7256a39a-061a-418e-b299-e67eacf4f780\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") "
Oct 30 07:45:22.805392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.785357    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\") pod \"7256a39a-061a-418e-b299-e67eacf4f780\" (UID: \"7256a39a-061a-418e-b299-e67eacf4f780\") "
Oct 30 07:45:22.805392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.786123    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pfr\" (UniqueName: \"kubernetes.io/projected/6222c938-39d6-4d25-b97a-22be3e24e33b-kube-api-access-f2pfr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:22.805392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.786140    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6222c938-39d6-4d25-b97a-22be3e24e33b-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:22.883373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:22.869583    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca" pod "7256a39a-061a-418e-b299-e67eacf4f780" (UID: "7256a39a-061a-418e-b299-e67eacf4f780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:23.024058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.023902    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256a39a-061a-418e-b299-e67eacf4f780-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.028612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.028470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:23.114656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.114504    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:23.114656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.114606    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889105-vmthx"]
Oct 30 07:45:23.150732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.139653    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8" pod "7256a39a-061a-418e-b299-e67eacf4f780" (UID: "7256a39a-061a-418e-b299-e67eacf4f780"). InnerVolumeSpecName "kube-api-access-tcmm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:23.150732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.143521    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmm8\" (UniqueName: \"kubernetes.io/projected/7256a39a-061a-418e-b299-e67eacf4f780-kube-api-access-tcmm8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.198381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.196849    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:23.246651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.246613    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\" (UniqueName: \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\") pod \"26cab399-d2e1-4535-95a8-af6877be8c00\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") "
Oct 30 07:45:23.246828 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.246718    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\") pod \"15fdb9b6-4633-4f67-881e-33f3bab8d110\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") "
Oct 30 07:45:23.246918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.246839    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\") pod \"26cab399-d2e1-4535-95a8-af6877be8c00\" (UID: \"26cab399-d2e1-4535-95a8-af6877be8c00\") "
Oct 30 07:45:23.247065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.247046    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\" (UniqueName: \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\") pod \"15fdb9b6-4633-4f67-881e-33f3bab8d110\" (UID: \"15fdb9b6-4633-4f67-881e-33f3bab8d110\") "
Oct 30 07:45:23.249477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.249440    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca" pod "26cab399-d2e1-4535-95a8-af6877be8c00" (UID: "26cab399-d2e1-4535-95a8-af6877be8c00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:23.250169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.250139    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca" pod "15fdb9b6-4633-4f67-881e-33f3bab8d110" (UID: "15fdb9b6-4633-4f67-881e-33f3bab8d110"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:23.251424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.251394    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:23.269832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.269789    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797" pod "15fdb9b6-4633-4f67-881e-33f3bab8d110" (UID: "15fdb9b6-4633-4f67-881e-33f3bab8d110"). InnerVolumeSpecName "kube-api-access-r8797". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:23.272551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.272520    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h" pod "26cab399-d2e1-4535-95a8-af6877be8c00" (UID: "26cab399-d2e1-4535-95a8-af6877be8c00"). InnerVolumeSpecName "kube-api-access-9vf7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:23.348421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.348385    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\" (UniqueName: \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\") pod \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") "
Oct 30 07:45:23.348585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.348456    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\") pod \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\" (UID: \"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f\") "
Oct 30 07:45:23.348866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.348837    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca" pod "f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" (UID: "f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:23.349303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.349277    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cab399-d2e1-4535-95a8-af6877be8c00-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.349349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.349314    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.349349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.349326    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8797\" (UniqueName: \"kubernetes.io/projected/15fdb9b6-4633-4f67-881e-33f3bab8d110-kube-api-access-r8797\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.349349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.349335    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vf7h\" (UniqueName: \"kubernetes.io/projected/26cab399-d2e1-4535-95a8-af6877be8c00-kube-api-access-9vf7h\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.349349 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.349345    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fdb9b6-4633-4f67-881e-33f3bab8d110-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.350703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.350673    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt" pod "f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" (UID: "f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f"). InnerVolumeSpecName "kube-api-access-gg4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:23.384361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.384321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv" event={"ID":"26cab399-d2e1-4535-95a8-af6877be8c00","Type":"ContainerDied","Data":"1486db05bd2049a387143cca0a01a2525356ded692730349b2f0fe4f49d4d572"}
Oct 30 07:45:23.384874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.384337    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"
Oct 30 07:45:23.384874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.384434    6738 scope.go:122] "RemoveContainer" containerID="0be993312d62fb940e73bc8545c81b6bdb905b77e98e264cd9faf5a3185ff884"
Oct 30 07:45:23.389608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.389453    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"
Oct 30 07:45:23.389608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.389459    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8" event={"ID":"6222c938-39d6-4d25-b97a-22be3e24e33b","Type":"ContainerDied","Data":"d2472f2cd503327d496105665ca62eea96a0293ff5cc19a542885e50c0e70ee8"}
Oct 30 07:45:23.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.399328    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k" event={"ID":"7256a39a-061a-418e-b299-e67eacf4f780","Type":"ContainerDied","Data":"14101fe55f747f0618c92e92f17895ec2e7c9dad24ff11e1af5cfba27215dfc6"}
Oct 30 07:45:23.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.399345    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"
Oct 30 07:45:23.400437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.399524    6738 scope.go:122] "RemoveContainer" containerID="73c05da480396918a14190cebada7dce891ccb2f513ce40a92b8bb5d53ee5a5b"
Oct 30 07:45:23.406884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.406848    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9" event={"ID":"f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f","Type":"ContainerDied","Data":"4e1515791610dc0f71bebc72809d08a9598b687db8b4b94e3ecf8fdabf32867f"}
Oct 30 07:45:23.406884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.406876    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"
Oct 30 07:45:23.413942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.413904    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"
Oct 30 07:45:23.414118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.413896    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp" event={"ID":"15fdb9b6-4633-4f67-881e-33f3bab8d110","Type":"ContainerDied","Data":"874851ae4a9e567c64c83133c5d1c1317473b4e7dbb97826ffb315e7c9d49b2f"}
Oct 30 07:45:23.418476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.418452    6738 scope.go:122] "RemoveContainer" containerID="2758490c805a55803b100c0cdfaaf17699c4ce1de9dc673bee77b5a704982656"
Oct 30 07:45:23.433649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.433626    6738 scope.go:122] "RemoveContainer" containerID="f61137e10cb8a1f190b55a09f5b55d132a04473f0f424cd877ce29dcab3702cd"
Oct 30 07:45:23.450895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.450675    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4nt\" (UniqueName: \"kubernetes.io/projected/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f-kube-api-access-gg4nt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:23.483049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.483019    6738 scope.go:122] "RemoveContainer" containerID="1fdccaf5c901b7be8ef2a7f579f6969cc679ac2bba079cc204656f1daa8d9071"
Oct 30 07:45:23.491036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.490936    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:23.497593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.497556    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889105-m7tv9"]
Oct 30 07:45:23.531202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.531170    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:23.535657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.535626    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889105-696w8"]
Oct 30 07:45:23.568102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.568061    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:23.572632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.572600    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4ab37e-29889105-ckk8k"]
Oct 30 07:45:23.605514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.605474    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:23.612627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.612589    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-ggxfv"]
Oct 30 07:45:23.652401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.652277    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:23.656997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.656964    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889105-hd7bp"]
Oct 30 07:45:23.693086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.693053    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" path="/var/lib/kubelet/pods/09e994a2-e628-419c-b0dd-58e677cdf9fc/volumes"
Oct 30 07:45:23.693657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.693641    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" path="/var/lib/kubelet/pods/15fdb9b6-4633-4f67-881e-33f3bab8d110/volumes"
Oct 30 07:45:23.694094 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.694081    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" path="/var/lib/kubelet/pods/160ab89d-32a5-47bd-b0a6-7dbb5d83407b/volumes"
Oct 30 07:45:23.694832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.694816    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" path="/var/lib/kubelet/pods/26cab399-d2e1-4535-95a8-af6877be8c00/volumes"
Oct 30 07:45:23.695335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.695320    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" path="/var/lib/kubelet/pods/3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c/volumes"
Oct 30 07:45:23.695762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.695749    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" path="/var/lib/kubelet/pods/6222c938-39d6-4d25-b97a-22be3e24e33b/volumes"
Oct 30 07:45:23.696486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.696471    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7256a39a-061a-418e-b299-e67eacf4f780" path="/var/lib/kubelet/pods/7256a39a-061a-418e-b299-e67eacf4f780/volumes"
Oct 30 07:45:23.696888 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.696875    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" path="/var/lib/kubelet/pods/79e20dd7-eea4-4d0d-a447-6d4d8afed2ed/volumes"
Oct 30 07:45:23.697288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.697276    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" path="/var/lib/kubelet/pods/83f358fa-509d-4e9f-bc0d-e0b5261a0ee9/volumes"
Oct 30 07:45:23.698015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:23.697998    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" path="/var/lib/kubelet/pods/f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f/volumes"
Oct 30 07:45:24.511646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.511605    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"]
Oct 30 07:45:24.512751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512717    6738 cpu_manager.go:414] "containerMap: removing container" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.512751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512747    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512765    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512772    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512781    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512788    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512797    6738 cpu_manager.go:414] "containerMap: removing container" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512803    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512812    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512818    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512843    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512850    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512874    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512881    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512890    6738 cpu_manager.go:414] "containerMap: removing container" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512897    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512919    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512926    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.512934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512936    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512944    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512954    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512961    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512970    6738 cpu_manager.go:414] "containerMap: removing container" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512977    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.512995    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513001    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513012    6738 cpu_manager.go:414] "containerMap: removing container" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513019    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513030    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513035    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513046    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513053    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513067    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513073    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513085    6738 cpu_manager.go:414] "containerMap: removing container" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513091    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513104    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513111    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513122    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513129    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513144    6738 cpu_manager.go:414] "containerMap: removing container" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513152    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513167    6738 cpu_manager.go:414] "containerMap: removing container" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513174    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513210    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513221    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513240    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513248    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513270    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513279    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513305    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513313    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513333    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513341    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513353    6738 cpu_manager.go:414] "containerMap: removing container" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513361    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513371    6738 cpu_manager.go:414] "containerMap: removing container" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513378    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513402    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513409    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513424    6738 cpu_manager.go:414] "containerMap: removing container" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513432    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513446    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513453    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513678    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513690    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513698    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513708    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513717    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513727    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513736    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513746    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513755    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513765    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513777    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513787    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513795    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513803    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513812    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513820    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513829    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513840    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513851    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513862    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513873    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513882    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513892    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513901    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:24.513870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513910    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513919    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513929    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513940    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513950    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513960    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513969    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513977    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:24.517264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.513985    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:24.520797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.520771    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.534948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.534915    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"]
Oct 30 07:45:24.541566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.541539    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:45:24.541738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.541648    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:45:24.568250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.568211    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.568519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.568307    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.568519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.568394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkt9z\" (UniqueName: \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.670080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.670033    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.670343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.670104    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.670343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.670133    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkt9z\" (UniqueName: \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.670924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.670890    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.672651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.672631    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.694165 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.694135    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkt9z\" (UniqueName: \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\") pod \"maas-api-revoke-sub-e073a368-zvzzm\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:24.840494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:24.840416    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:25.029738 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:25.029703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:25.190507 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:25.190469    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c243912_ba27_4d8f_919b_ef256772a49b.slice/crio-e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e WatchSource:0}: Error finding container e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e: Status 404 returned error can't find the container with id e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e
Oct 30 07:45:25.191318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:25.191267    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"]
Oct 30 07:45:25.441339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:25.441217    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm" event={"ID":"6c243912-ba27-4d8f-919b-ef256772a49b","Type":"ContainerStarted","Data":"e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e"}
Oct 30 07:45:26.032580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.032536    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"]
Oct 30 07:45:26.045460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.045422    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b99658-dqqdx"]
Oct 30 07:45:26.049958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.049915    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"]
Oct 30 07:45:26.054217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.054184    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28ce42e-rmczt"]
Oct 30 07:45:26.454676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.454641    6738 generic.go:356] "Generic (PLEG): container finished" podID="6c243912-ba27-4d8f-919b-ef256772a49b" containerID="4d611f5a3c46f0c6448c6cfec0c0df3414d8cf2e405413d2d8a16e0ef7581462" exitCode=0
Oct 30 07:45:26.454856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.454727    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm" event={"ID":"6c243912-ba27-4d8f-919b-ef256772a49b","Type":"ContainerDied","Data":"4d611f5a3c46f0c6448c6cfec0c0df3414d8cf2e405413d2d8a16e0ef7581462"}
Oct 30 07:45:26.859464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:26.859382    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:27.636317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.636277    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:27.696230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.696197    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793decc0-4eb7-4bd6-a03a-81a97a2b2549" path="/var/lib/kubelet/pods/793decc0-4eb7-4bd6-a03a-81a97a2b2549/volumes"
Oct 30 07:45:27.696795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.696776    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0916e46-f23a-455b-ac08-f90f271f7c09" path="/var/lib/kubelet/pods/c0916e46-f23a-455b-ac08-f90f271f7c09/volumes"
Oct 30 07:45:27.760798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.760705    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\" (UniqueName: \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\") pod \"6c243912-ba27-4d8f-919b-ef256772a49b\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") "
Oct 30 07:45:27.760952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.760893    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\") pod \"6c243912-ba27-4d8f-919b-ef256772a49b\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") "
Oct 30 07:45:27.761015 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.760958    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\") pod \"6c243912-ba27-4d8f-919b-ef256772a49b\" (UID: \"6c243912-ba27-4d8f-919b-ef256772a49b\") "
Oct 30 07:45:27.761419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.761387    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca" pod "6c243912-ba27-4d8f-919b-ef256772a49b" (UID: "6c243912-ba27-4d8f-919b-ef256772a49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:27.761942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.761909    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:27.763003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.762979    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z" pod "6c243912-ba27-4d8f-919b-ef256772a49b" (UID: "6c243912-ba27-4d8f-919b-ef256772a49b"). InnerVolumeSpecName "kube-api-access-dkt9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:27.763090 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.763057    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca" pod "6c243912-ba27-4d8f-919b-ef256772a49b" (UID: "6c243912-ba27-4d8f-919b-ef256772a49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:45:27.863201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.863157    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkt9z\" (UniqueName: \"kubernetes.io/projected/6c243912-ba27-4d8f-919b-ef256772a49b-kube-api-access-dkt9z\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:27.863201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:27.863197    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c243912-ba27-4d8f-919b-ef256772a49b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:28.475709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:28.475676    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"
Oct 30 07:45:28.475955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:28.475679    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm" event={"ID":"6c243912-ba27-4d8f-919b-ef256772a49b","Type":"ContainerDied","Data":"e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e"}
Oct 30 07:45:28.475955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:28.475803    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62d274af587c6ef1ce21917694468019d3096921d614adc3212462909f3f72e"
Oct 30 07:45:28.777001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:28.776849    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:29.908830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:29.908795    6738 scope.go:122] "RemoveContainer" containerID="e769f9ef5e23aa5cbf2f93c1db86460e31db6de75a9a9f6bf360ba058e12b7ee"
Oct 30 07:45:29.920811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:29.920785    6738 scope.go:122] "RemoveContainer" containerID="96384e511c07e312caff4aab5f1519db08771a287ae0f4900053eed7dc98458b"
Oct 30 07:45:29.932632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:29.932605    6738 scope.go:122] "RemoveContainer" containerID="6836fd0d1ee0c567478598f451dae4cb7c46fb5c59f8809f0b4473780690f0d3"
Oct 30 07:45:30.033131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:30.033098    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"]
Oct 30 07:45:30.046032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:30.046007    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1d4092-7d25j"]
Oct 30 07:45:30.616713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:30.616675    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:31.692730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:31.692690    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b1ff38-5680-4315-852c-c7a98c922b4b" path="/var/lib/kubelet/pods/18b1ff38-5680-4315-852c-c7a98c922b4b/volumes"
Oct 30 07:45:32.466528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:32.466498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:33.506075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:33.506040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:34.354820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:34.354781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:35.318839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.318806    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"]
Oct 30 07:45:35.319919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319893    6738 cpu_manager.go:414] "containerMap: removing container" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:35.319919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319919    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319943    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319950    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319976    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319984    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.319997    6738 cpu_manager.go:414] "containerMap: removing container" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320005    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320020    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320027    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320043    6738 cpu_manager.go:414] "containerMap: removing container" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320049    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cab399-d2e1-4535-95a8-af6877be8c00" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320075    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320082    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320108    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320115    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7256a39a-061a-418e-b299-e67eacf4f780" containerName="cleanup"
Oct 30 07:45:35.320132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320131    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320139    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320152    6738 cpu_manager.go:414] "containerMap: removing container" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320160    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320187    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320193    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6222c938-39d6-4d25-b97a-22be3e24e33b" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320210    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6c243912-ba27-4d8f-919b-ef256772a49b" containerName="revoke-keys"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320217    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c243912-ba27-4d8f-919b-ef256772a49b" containerName="revoke-keys"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320230    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320237    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320251    6738 cpu_manager.go:414] "containerMap: removing container" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320258    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320479    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5096ae27-38bc-4d4e-ad83-1efcaf982889" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320496    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f358fa-509d-4e9f-bc0d-e0b5261a0ee9" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320510    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4e0dcf-0f6f-4ba7-8b7e-e3cd72150c1c" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320521    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="160ab89d-32a5-47bd-b0a6-7dbb5d83407b" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320534    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f14fbb-46ac-4ac1-8b1f-50d8841fce1f" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320545    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5d32a9-515b-415f-85a7-45d6e50d2c18" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320559    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e20dd7-eea4-4d0d-a447-6d4d8afed2ed" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320573    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c243912-ba27-4d8f-919b-ef256772a49b" containerName="revoke-keys"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320586    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12e79b-7d3f-4ee9-96c6-4d6d8533d299" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320596    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fdb9b6-4633-4f67-881e-33f3bab8d110" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320607    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73a2fe-05e8-4675-ab21-f422ea65b18f" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320619    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfab278-76ce-4cdb-869e-c1c7b192c490" containerName="cleanup"
Oct 30 07:45:35.320896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.320629    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e994a2-e628-419c-b0dd-58e677cdf9fc" containerName="cleanup"
Oct 30 07:45:35.327414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.327391    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.342805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.342774    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"]
Oct 30 07:45:35.348288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.348260    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:45:35.348625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.348594    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:45:35.393801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.393752    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclxq\" (UniqueName: \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.393985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.393823    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.394078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.394056    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.495577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.495542    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclxq\" (UniqueName: \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.495760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.495595    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.495823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.495756    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.496462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.496435    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.498306 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.498270    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.519866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.519827    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclxq\" (UniqueName: \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\") pod \"maas-api-revoke-sub-0d174c9a-vwgjl\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.648896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.648853    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:35.976599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:35.976565    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"]
Oct 30 07:45:35.978167 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:35.978135    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ddf8c4_0fcb_41ae_9a52_eef5bbb7bbab.slice/crio-d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e WatchSource:0}: Error finding container d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e: Status 404 returned error can't find the container with id d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e
Oct 30 07:45:36.241570 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:36.241478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:36.597838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:36.597752    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl" event={"ID":"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab","Type":"ContainerStarted","Data":"5342e8629a52975b3ac214ccaac38ae24f31fb782cd50391f78c711b26e8ca17"}
Oct 30 07:45:36.597838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:36.597806    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl" event={"ID":"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab","Type":"ContainerStarted","Data":"d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e"}
Oct 30 07:45:36.639778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:36.639707    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl" podStartSLOduration=1.639689095 podStartE2EDuration="1.639689095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:36.638960521 +0000 UTC m=+2897.071724414" watchObservedRunningTime="2026-10-30 07:45:36.639689095 +0000 UTC m=+2897.072452992"
Oct 30 07:45:37.520273 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:45:37.520235    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:45:37.609351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:37.609286    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" containerID="5342e8629a52975b3ac214ccaac38ae24f31fb782cd50391f78c711b26e8ca17" exitCode=0
Oct 30 07:45:37.609795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:37.609397    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl" event={"ID":"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab","Type":"ContainerDied","Data":"5342e8629a52975b3ac214ccaac38ae24f31fb782cd50391f78c711b26e8ca17"}
Oct 30 07:45:38.205451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.205416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:38.803056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.803026    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:38.853533 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.853497    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\" (UniqueName: \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\") pod \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") "
Oct 30 07:45:38.853756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.853594    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\") pod \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") "
Oct 30 07:45:38.853756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.853672    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\") pod \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\" (UID: \"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab\") "
Oct 30 07:45:38.854671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.854643    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca" pod "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" (UID: "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:38.856594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.856570    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca" pod "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" (UID: "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:45:38.860810 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.860781    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq" pod "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" (UID: "f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab"). InnerVolumeSpecName "kube-api-access-sclxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:38.956362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.956328    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclxq\" (UniqueName: \"kubernetes.io/projected/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-kube-api-access-sclxq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:38.956362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.956358    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:38.956362 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:38.956368    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:39.631371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:39.631325    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl" event={"ID":"f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab","Type":"ContainerDied","Data":"d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e"}
Oct 30 07:45:39.631371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:39.631349    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"
Oct 30 07:45:39.631598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:39.631384    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70c53303c1964bf560667791b3454fe945b6a1efb6750ba974bda121a42c07e"
Oct 30 07:45:40.028971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:40.028855    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:40.148328 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:40.148276    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:40.593787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:40.593691    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:42.022180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:42.022135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:43.984976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:43.984935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:44.109474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:44.109436    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:44.229919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:44.229881    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:46.163229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:46.163187    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:47.042592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.042549    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:47.186375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.183895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:47.512527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.512486    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:45:47.594153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.594115    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:47.752130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.752096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:47.942315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:47.941692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:48.415122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:48.415073    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:48.574617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:48.574581    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:45:48.656535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:48.656495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:50.354334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:50.354283    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:51.066303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.066264    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"]
Oct 30 07:45:51.067266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.067244    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" containerName="revoke-keys"
Oct 30 07:45:51.067266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.067267    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" containerName="revoke-keys"
Oct 30 07:45:51.067526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.067509    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" containerName="revoke-keys"
Oct 30 07:45:51.071831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.071810    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.089519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.089486    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"]
Oct 30 07:45:51.096023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.095991    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:45:51.096225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.096098    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:45:51.151139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.151106    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.151372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.151267    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.151443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.151425    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlq5\" (UniqueName: \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.253214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.253172    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlq5\" (UniqueName: \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.253450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.253389    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.253529 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.253488    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.254066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.254042    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.255952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.255930    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.276993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.276958    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlq5\" (UniqueName: \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\") pod \"maas-api-revoke-sub-40183da1-z49mm\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.389158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.389123    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:51.699320 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:51.699271    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2042562_74a6_496c_8524_3b80d8892414.slice/crio-adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec WatchSource:0}: Error finding container adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec: Status 404 returned error can't find the container with id adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec
Oct 30 07:45:51.700831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.700802    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"]
Oct 30 07:45:51.752932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:51.752896    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm" event={"ID":"f2042562-74a6-496c-8524-3b80d8892414","Type":"ContainerStarted","Data":"adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec"}
Oct 30 07:45:52.125854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.125820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:52.513183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.513144    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" podUID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" containerName="payload-processing" containerID="cri-o://24a9e1715361cccfa28f0fccf2e19f2209a4cced9f6b47e1af4e4206402771c1" gracePeriod=25
Oct 30 07:45:52.783068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.782971    6738 generic.go:356] "Generic (PLEG): container finished" podID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" containerID="24a9e1715361cccfa28f0fccf2e19f2209a4cced9f6b47e1af4e4206402771c1" exitCode=0
Oct 30 07:45:52.783068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.783047    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" event={"ID":"aff7d69f-11f1-4f9b-aa29-eeda025cadf9","Type":"ContainerDied","Data":"24a9e1715361cccfa28f0fccf2e19f2209a4cced9f6b47e1af4e4206402771c1"}
Oct 30 07:45:52.787857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.787449    6738 generic.go:356] "Generic (PLEG): container finished" podID="f2042562-74a6-496c-8524-3b80d8892414" containerID="976adf9fa7b799b9337b596f9723bf7d44c8a060af1dda5f0e71c3a2ee611e10" exitCode=0
Oct 30 07:45:52.787857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.787496    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm" event={"ID":"f2042562-74a6-496c-8524-3b80d8892414","Type":"ContainerDied","Data":"976adf9fa7b799b9337b596f9723bf7d44c8a060af1dda5f0e71c3a2ee611e10"}
Oct 30 07:45:52.859140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.859108    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:45:52.913098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913045    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\") pod \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") "
Oct 30 07:45:52.913363 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913195    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\") pod \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") "
Oct 30 07:45:52.913459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913374    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\") pod \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") "
Oct 30 07:45:52.913518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913477    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\") pod \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\" (UID: \"aff7d69f-11f1-4f9b-aa29-eeda025cadf9\") "
Oct 30 07:45:52.913637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913610    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume" pod "aff7d69f-11f1-4f9b-aa29-eeda025cadf9" (UID: "aff7d69f-11f1-4f9b-aa29-eeda025cadf9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:52.913753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.913728    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume" pod "aff7d69f-11f1-4f9b-aa29-eeda025cadf9" (UID: "aff7d69f-11f1-4f9b-aa29-eeda025cadf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:52.914711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.914676    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:52.914711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.914705    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:52.915954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.915926    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw" pod "aff7d69f-11f1-4f9b-aa29-eeda025cadf9" (UID: "aff7d69f-11f1-4f9b-aa29-eeda025cadf9"). InnerVolumeSpecName "kube-api-access-f5ddw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:52.916261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:52.916240    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials" pod "aff7d69f-11f1-4f9b-aa29-eeda025cadf9" (UID: "aff7d69f-11f1-4f9b-aa29-eeda025cadf9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:53.015781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.015740    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:53.015781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.015772    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ddw\" (UniqueName: \"kubernetes.io/projected/aff7d69f-11f1-4f9b-aa29-eeda025cadf9-kube-api-access-f5ddw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:53.575280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.575188    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" podUID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" containerName="payload-processing" containerID="cri-o://28ca196b114eead683e019d31158e365a727d4303aacad5faf4cff4c744a9dd4" gracePeriod=25
Oct 30 07:45:53.800414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.800380    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"
Oct 30 07:45:53.800635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.800357    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn" event={"ID":"aff7d69f-11f1-4f9b-aa29-eeda025cadf9","Type":"ContainerDied","Data":"ec64155ad4d8be2fb639b5abce7a662221d2ac1ac9803366917976a404f923f8"}
Oct 30 07:45:53.800635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.800606    6738 scope.go:122] "RemoveContainer" containerID="24a9e1715361cccfa28f0fccf2e19f2209a4cced9f6b47e1af4e4206402771c1"
Oct 30 07:45:53.802763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.802735    6738 generic.go:356] "Generic (PLEG): container finished" podID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" containerID="28ca196b114eead683e019d31158e365a727d4303aacad5faf4cff4c744a9dd4" exitCode=0
Oct 30 07:45:53.802890 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.802803    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" event={"ID":"72500675-1c7e-4ca0-9c35-22428f8eb1b9","Type":"ContainerDied","Data":"28ca196b114eead683e019d31158e365a727d4303aacad5faf4cff4c744a9dd4"}
Oct 30 07:45:53.859194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.858374    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:45:53.873102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.873073    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-85477mshn"]
Oct 30 07:45:53.908642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.908609    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:45:53.930131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930099    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\") pod \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") "
Oct 30 07:45:53.930361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930164    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\") pod \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") "
Oct 30 07:45:53.930361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930202    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\") pod \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") "
Oct 30 07:45:53.930559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930443    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\") pod \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\" (UID: \"72500675-1c7e-4ca0-9c35-22428f8eb1b9\") "
Oct 30 07:45:53.930678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930652    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume" pod "72500675-1c7e-4ca0-9c35-22428f8eb1b9" (UID: "72500675-1c7e-4ca0-9c35-22428f8eb1b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:53.931022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.930965    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume" pod "72500675-1c7e-4ca0-9c35-22428f8eb1b9" (UID: "72500675-1c7e-4ca0-9c35-22428f8eb1b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:53.931183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.931165    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:53.931243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.931186    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72500675-1c7e-4ca0-9c35-22428f8eb1b9-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:53.933731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.933700    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials" pod "72500675-1c7e-4ca0-9c35-22428f8eb1b9" (UID: "72500675-1c7e-4ca0-9c35-22428f8eb1b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:53.933848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.933737    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq" pod "72500675-1c7e-4ca0-9c35-22428f8eb1b9" (UID: "72500675-1c7e-4ca0-9c35-22428f8eb1b9"). InnerVolumeSpecName "kube-api-access-56qdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:53.983410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:53.983377    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:54.032002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.031969    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\") pod \"f2042562-74a6-496c-8524-3b80d8892414\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") "
Oct 30 07:45:54.032224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032106    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\" (UniqueName: \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\") pod \"f2042562-74a6-496c-8524-3b80d8892414\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") "
Oct 30 07:45:54.032224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032165    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\") pod \"f2042562-74a6-496c-8524-3b80d8892414\" (UID: \"f2042562-74a6-496c-8524-3b80d8892414\") "
Oct 30 07:45:54.032621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032591    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca" pod "f2042562-74a6-496c-8524-3b80d8892414" (UID: "f2042562-74a6-496c-8524-3b80d8892414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:54.032787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032764    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:54.032787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032789    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qdq\" (UniqueName: \"kubernetes.io/projected/72500675-1c7e-4ca0-9c35-22428f8eb1b9-kube-api-access-56qdq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:54.032988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.032804    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2042562-74a6-496c-8524-3b80d8892414-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:54.034251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.034224    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca" pod "f2042562-74a6-496c-8524-3b80d8892414" (UID: "f2042562-74a6-496c-8524-3b80d8892414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:45:54.034369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.034278    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5" pod "f2042562-74a6-496c-8524-3b80d8892414" (UID: "f2042562-74a6-496c-8524-3b80d8892414"). InnerVolumeSpecName "kube-api-access-vrlq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:54.058153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.058106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:54.160200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.159488    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlq5\" (UniqueName: \"kubernetes.io/projected/f2042562-74a6-496c-8524-3b80d8892414-kube-api-access-vrlq5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:54.160200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.159583    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2042562-74a6-496c-8524-3b80d8892414-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:45:54.523280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.523184    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:54.640351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.640269    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:54.814524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.814437    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"
Oct 30 07:45:54.814524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.814464    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9" event={"ID":"72500675-1c7e-4ca0-9c35-22428f8eb1b9","Type":"ContainerDied","Data":"5c2dbc29feb0c6fdd6e2a9a71e0dd17dbef9bf8c4b8dd49adb3a9e4316fc67d8"}
Oct 30 07:45:54.814765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.814532    6738 scope.go:122] "RemoveContainer" containerID="28ca196b114eead683e019d31158e365a727d4303aacad5faf4cff4c744a9dd4"
Oct 30 07:45:54.817500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.817476    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"
Oct 30 07:45:54.817500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.817479    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm" event={"ID":"f2042562-74a6-496c-8524-3b80d8892414","Type":"ContainerDied","Data":"adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec"}
Oct 30 07:45:54.817702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.817531    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adac5cb88a683dbe34c66d6aee213795c6d989f0c38c1fa003aa9ce076d354ec"
Oct 30 07:45:54.869734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.869700    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:45:54.881964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:54.881928    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8f1df/payload-processing-external-model-e2e-extmod-a-a8f1df-749cnfmw9"]
Oct 30 07:45:55.031244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.031201    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:55.432634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.432597    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:45:55.433973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.433940    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f2042562-74a6-496c-8524-3b80d8892414" containerName="revoke-keys"
Oct 30 07:45:55.433973 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.433969    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2042562-74a6-496c-8524-3b80d8892414" containerName="revoke-keys"
Oct 30 07:45:55.434194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434033    6738 cpu_manager.go:414] "containerMap: removing container" podUID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" containerName="payload-processing"
Oct 30 07:45:55.434194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434045    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" containerName="payload-processing"
Oct 30 07:45:55.434194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434066    6738 cpu_manager.go:414] "containerMap: removing container" podUID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" containerName="payload-processing"
Oct 30 07:45:55.434194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434075    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" containerName="payload-processing"
Oct 30 07:45:55.434468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434448    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2042562-74a6-496c-8524-3b80d8892414" containerName="revoke-keys"
Oct 30 07:45:55.434532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434474    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" containerName="payload-processing"
Oct 30 07:45:55.434532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.434486    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" containerName="payload-processing"
Oct 30 07:45:55.440940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.440914    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.457817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.457789    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:45:55.464118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.464086    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ccac6f-openshift-default-dockercfg-f8r2g\""
Oct 30 07:45:55.484283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.484248    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587416    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587517    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587545    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587609    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587769    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587844    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.587898    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.588020    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.588315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.588050    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzz89\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.692943    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.692988    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzz89\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693053    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693115    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693142    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693201    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693365    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693438    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693484    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.693977    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.697370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.694245    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.708227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.708189    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.708653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.708627    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.709892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.709839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.728847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.728063    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.729623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.729590    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72500675-1c7e-4ca0-9c35-22428f8eb1b9" path="/var/lib/kubelet/pods/72500675-1c7e-4ca0-9c35-22428f8eb1b9/volumes"
Oct 30 07:45:55.730572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.730549    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff7d69f-11f1-4f9b-aa29-eeda025cadf9" path="/var/lib/kubelet/pods/aff7d69f-11f1-4f9b-aa29-eeda025cadf9/volumes"
Oct 30 07:45:55.737413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.737387    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.789171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.789135    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzz89\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.832170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.832103    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\") pod \"e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") " pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:55.935079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:55.933690    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:56.058081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.057990    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:56.158172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.158133    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:56.387820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.387790    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:45:56.388551 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:45:56.388523    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b0a9e9_6fcd_476a_9436_61efa2e9d1b1.slice/crio-d5a76b0ecba1de444753368083be083e7ab29ddc9853b3dfef6f80492610978d WatchSource:0}: Error finding container d5a76b0ecba1de444753368083be083e7ab29ddc9853b3dfef6f80492610978d: Status 404 returned error can't find the container with id d5a76b0ecba1de444753368083be083e7ab29ddc9853b3dfef6f80492610978d
Oct 30 07:45:56.391899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.391860    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:45:56.392007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.391931    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:45:56.392007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.391961    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:45:56.860233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.860146    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerStarted","Data":"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"}
Oct 30 07:45:56.860233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.860203    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerStarted","Data":"d5a76b0ecba1de444753368083be083e7ab29ddc9853b3dfef6f80492610978d"}
Oct 30 07:45:56.923692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:56.923633    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podStartSLOduration=1.923617181 podStartE2EDuration="1.923617181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:56.918880977 +0000 UTC m=+2917.351644870" watchObservedRunningTime="2026-10-30 07:45:56.923617181 +0000 UTC m=+2917.356381146"
Oct 30 07:45:57.068704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.068648    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:57.092327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.091438    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:57.772592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.772546    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:45:57.772834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.772816    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" containerID="cri-o://ab29800fdfb76a0870e0ddfa63bf2adcedc90bef01b0fa3deb3ec19410bf4ae4" gracePeriod=30
Oct 30 07:45:57.868285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.868253    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:57.869365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:57.869345    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:45:58.053213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:58.053120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:59.272899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:59.272864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:45:59.402153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:45:59.402116    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:00.189878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:00.189846    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:01.049688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.049652    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:01.198370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.198278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:01.210255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.210221    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:46:01.216234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.216205    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.239995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.239958    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ccac6f-dockercfg-r9wks\""
Oct 30 07:46:01.257682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.257650    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:46:01.284176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.284139    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ccac6f\""
Oct 30 07:46:01.460320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.402270    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.460320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.402630    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99kqg\" (UniqueName: \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.636585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.636540    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kqg\" (UniqueName: \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.636794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.636681    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.637501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.637481    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.660732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.660692    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kqg\" (UniqueName: \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\") pod \"payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:01.761219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.761125    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:46:01.761219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.761188    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:46:01.848395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:01.848358    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:02.164897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.163085    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"]
Oct 30 07:46:02.182909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.182874    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.209203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.209174    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"]
Oct 30 07:46:02.224603 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:02.224563    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d94de_6a65_4372_b772_bd65f09cae53.slice/crio-a5eb78b27ac58064499fac64dc7b6f390e995c3f672738bae93a9b3915180b3c WatchSource:0}: Error finding container a5eb78b27ac58064499fac64dc7b6f390e995c3f672738bae93a9b3915180b3c: Status 404 returned error can't find the container with id a5eb78b27ac58064499fac64dc7b6f390e995c3f672738bae93a9b3915180b3c
Oct 30 07:46:02.225914 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.225890    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:46:02.236668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.236637    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:46:02.239778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.239749    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:46:02.354922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.354882    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.355107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.355054    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.355249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.355236    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qk2\" (UniqueName: \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.457180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.457082    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.457406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.457218    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.457484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.457436    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qk2\" (UniqueName: \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.457823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.457799    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.459717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.459694    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.474196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.474164    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:02.481089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.481062    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qk2\" (UniqueName: \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\") pod \"maas-api-revoke-sub-aa90ed1f-dpl6j\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.534341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.527095    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:02.875049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.874998    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"]
Oct 30 07:46:02.884668 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:02.884636    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c64aa94_7e8f_4e26_919d_234e9df628a1.slice/crio-52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e WatchSource:0}: Error finding container 52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e: Status 404 returned error can't find the container with id 52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e
Oct 30 07:46:02.915511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.915473    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j" event={"ID":"8c64aa94-7e8f-4e26-919d-234e9df628a1","Type":"ContainerStarted","Data":"52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e"}
Oct 30 07:46:02.917958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.917915    6738 generic.go:356] "Generic (PLEG): container finished" podID="882b7b57-7108-4428-9ab0-5a5397226c00" containerID="ab29800fdfb76a0870e0ddfa63bf2adcedc90bef01b0fa3deb3ec19410bf4ae4" exitCode=0
Oct 30 07:46:02.918125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.917986    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" event={"ID":"882b7b57-7108-4428-9ab0-5a5397226c00","Type":"ContainerDied","Data":"ab29800fdfb76a0870e0ddfa63bf2adcedc90bef01b0fa3deb3ec19410bf4ae4"}
Oct 30 07:46:02.920077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.920052    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" event={"ID":"ec2d94de-6a65-4372-b772-bd65f09cae53","Type":"ContainerStarted","Data":"746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b"}
Oct 30 07:46:02.920215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.920101    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" event={"ID":"ec2d94de-6a65-4372-b772-bd65f09cae53","Type":"ContainerStarted","Data":"a5eb78b27ac58064499fac64dc7b6f390e995c3f672738bae93a9b3915180b3c"}
Oct 30 07:46:02.920215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.920204    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:02.977403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:02.977334    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" podStartSLOduration=1.977311539 podStartE2EDuration="1.977311539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:02.964244809 +0000 UTC m=+2923.397008706" watchObservedRunningTime="2026-10-30 07:46:02.977311539 +0000 UTC m=+2923.410075429"
Oct 30 07:46:03.127352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.127319    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:46:03.211576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211543    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211629    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211675    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211748    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211802    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.211892    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.212109    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.212200    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.212244    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\") pod \"882b7b57-7108-4428-9ab0-5a5397226c00\" (UID: \"882b7b57-7108-4428-9ab0-5a5397226c00\") "
Oct 30 07:46:03.212935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.212905    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:03.213198 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.213053    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:03.213321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.213270    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:03.213393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.213180    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:03.213393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.213281    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:03.214417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.214394    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:46:03.214911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.214892    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:03.215624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.215601    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:03.216795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.216774    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k" pod "882b7b57-7108-4428-9ab0-5a5397226c00" (UID: "882b7b57-7108-4428-9ab0-5a5397226c00"). InnerVolumeSpecName "kube-api-access-xlr9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:03.315013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.314977    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315009    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315021    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882b7b57-7108-4428-9ab0-5a5397226c00-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315032    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315044    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315054    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlr9k\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-kube-api-access-xlr9k\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315062    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882b7b57-7108-4428-9ab0-5a5397226c00-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315071    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882b7b57-7108-4428-9ab0-5a5397226c00-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.315353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.315079    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882b7b57-7108-4428-9ab0-5a5397226c00-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:03.931950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.931912    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j" event={"ID":"8c64aa94-7e8f-4e26-919d-234e9df628a1","Type":"ContainerStarted","Data":"89813a209e779e2ea086afde14a0527ea7061af4b077f2b0b446bd8210598cad"}
Oct 30 07:46:03.933556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.933534    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"
Oct 30 07:46:03.933722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.933573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d" event={"ID":"882b7b57-7108-4428-9ab0-5a5397226c00","Type":"ContainerDied","Data":"08fc4fce07554e23b4c8da28e075f01007ec7ad9c6acd2ab562456187364f41f"}
Oct 30 07:46:03.933722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.933623    6738 scope.go:122] "RemoveContainer" containerID="ab29800fdfb76a0870e0ddfa63bf2adcedc90bef01b0fa3deb3ec19410bf4ae4"
Oct 30 07:46:03.974401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:03.974342    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j" podStartSLOduration=1.9743248389999999 podStartE2EDuration="1.974324839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:03.972703787 +0000 UTC m=+2924.405467681" watchObservedRunningTime="2026-10-30 07:46:03.974324839 +0000 UTC m=+2924.407088734"
Oct 30 07:46:04.015528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:04.015491    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:46:04.025639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:04.025605    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xmv7d"]
Oct 30 07:46:04.436909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:04.436873    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:04.944215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:04.944176    6738 generic.go:356] "Generic (PLEG): container finished" podID="8c64aa94-7e8f-4e26-919d-234e9df628a1" containerID="89813a209e779e2ea086afde14a0527ea7061af4b077f2b0b446bd8210598cad" exitCode=0
Oct 30 07:46:04.944447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:04.944258    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j" event={"ID":"8c64aa94-7e8f-4e26-919d-234e9df628a1","Type":"ContainerDied","Data":"89813a209e779e2ea086afde14a0527ea7061af4b077f2b0b446bd8210598cad"}
Oct 30 07:46:05.693287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:05.693251    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" path="/var/lib/kubelet/pods/882b7b57-7108-4428-9ab0-5a5397226c00/volumes"
Oct 30 07:46:06.140603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.140574    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:06.210543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.210442    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:06.284953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.284909    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\") pod \"8c64aa94-7e8f-4e26-919d-234e9df628a1\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") "
Oct 30 07:46:06.285217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.284966    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\" (UniqueName: \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\") pod \"8c64aa94-7e8f-4e26-919d-234e9df628a1\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") "
Oct 30 07:46:06.285217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.285072    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\") pod \"8c64aa94-7e8f-4e26-919d-234e9df628a1\" (UID: \"8c64aa94-7e8f-4e26-919d-234e9df628a1\") "
Oct 30 07:46:06.296182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.292859    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca" pod "8c64aa94-7e8f-4e26-919d-234e9df628a1" (UID: "8c64aa94-7e8f-4e26-919d-234e9df628a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:46:06.301319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.298552    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca" pod "8c64aa94-7e8f-4e26-919d-234e9df628a1" (UID: "8c64aa94-7e8f-4e26-919d-234e9df628a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:06.310671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.310325    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2" pod "8c64aa94-7e8f-4e26-919d-234e9df628a1" (UID: "8c64aa94-7e8f-4e26-919d-234e9df628a1"). InnerVolumeSpecName "kube-api-access-54qk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:06.390668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.390631    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:06.390668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.390665    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qk2\" (UniqueName: \"kubernetes.io/projected/8c64aa94-7e8f-4e26-919d-234e9df628a1-kube-api-access-54qk2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:06.390668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.390678    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c64aa94-7e8f-4e26-919d-234e9df628a1-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:06.971138 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.971109    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"
Oct 30 07:46:06.971582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.971143    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j" event={"ID":"8c64aa94-7e8f-4e26-919d-234e9df628a1","Type":"ContainerDied","Data":"52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e"}
Oct 30 07:46:06.971582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:06.971197    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52fe0cd72a32be1c1067adb5db0862d037a02fa639b1ddc5733e27f630b7f13e"
Oct 30 07:46:07.521013 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:07.520971    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:46:08.058932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:08.058883    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:09.831453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:09.831418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:11.619559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:11.619519    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:13.427141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:13.427107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:13.937239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:13.937209    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:46:14.495374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.495336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:14.567339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.567270    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:14.568170 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568152    6738 cpu_manager.go:414] "containerMap: removing container" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy"
Oct 30 07:46:14.568220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568171    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy"
Oct 30 07:46:14.568220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568212    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8c64aa94-7e8f-4e26-919d-234e9df628a1" containerName="revoke-keys"
Oct 30 07:46:14.568220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568217    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c64aa94-7e8f-4e26-919d-234e9df628a1" containerName="revoke-keys"
Oct 30 07:46:14.568386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568375    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="882b7b57-7108-4428-9ab0-5a5397226c00" containerName="istio-proxy"
Oct 30 07:46:14.568424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.568396    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c64aa94-7e8f-4e26-919d-234e9df628a1" containerName="revoke-keys"
Oct 30 07:46:14.572812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.572789    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.589453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.589423    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:14.658676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.658638    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk25\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.658868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.658684    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.658868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.658793    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.658868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.658826    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.752434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.752347    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"]
Oct 30 07:46:14.759027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.758984    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.762525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.762482    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.762795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.762774    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.763231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.763210    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk25\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.763403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.763385    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.764501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.764477    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.765516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.765493    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.767708 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.767170    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.778754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.778717    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"]
Oct 30 07:46:14.783591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.783562    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:46:14.783778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.783562    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:46:14.891344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.869553    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.891344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.869658    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.891344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.869904    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.933329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.928031    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk25\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\") pod \"payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") " pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:14.979318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.976474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.979318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.976672    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.979318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.976764    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:14.979318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:14.978060    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:15.002564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.002414    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:15.052181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.052143    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a198e6-7tdqg\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:15.102218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.102173    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:15.190988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.190663    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:15.588911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.564274    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"]
Oct 30 07:46:15.593924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.593884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:15.791791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.791721    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:15.797411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.797387    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:15.815820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.815782    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:15.856202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.856164    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:15.932321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.923591    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:15.932321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.923691    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8d7r\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:15.932321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.923757    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:15.932321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:15.923943    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.025493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.025454    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.025706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.025544    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8d7r\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.025706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.025613    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.025849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.025784    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.026304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.026273    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.026445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.026275    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.028415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.028392    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.049740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.049662    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8d7r\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.066483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.066451    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" event={"ID":"d610984a-46e3-4855-b8a8-76863d2be9ff","Type":"ContainerStarted","Data":"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"}
Oct 30 07:46:16.066676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.066512    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" event={"ID":"d610984a-46e3-4855-b8a8-76863d2be9ff","Type":"ContainerStarted","Data":"e663ed30beb41ed5cbb34da42c027a7503f1ab5eed17e62ef64aa5a9ccecaea7"}
Oct 30 07:46:16.066676 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.066539    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:16.068025 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.067993    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg" event={"ID":"534f9eef-4c59-456b-bd56-9eeac8990c24","Type":"ContainerStarted","Data":"fa49839f0c82aa121664b38601aab4d70b50411d14797a5d568c95c7f9cd35a7"}
Oct 30 07:46:16.108285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.108231    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" podStartSLOduration=2.1082139030000002 podStartE2EDuration="2.108213903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:16.107912144 +0000 UTC m=+2936.540676037" watchObservedRunningTime="2026-10-30 07:46:16.108213903 +0000 UTC m=+2936.540977796"
Oct 30 07:46:16.133364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.133328    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:16.447221 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:16.447182    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32661d93_ae0a_41c3_bcfc_f0d7be94433a.slice/crio-cd08e51a3ed8545e038bc311398b0903a250eae3d8a46e3acedfc20420c888d2 WatchSource:0}: Error finding container cd08e51a3ed8545e038bc311398b0903a250eae3d8a46e3acedfc20420c888d2: Status 404 returned error can't find the container with id cd08e51a3ed8545e038bc311398b0903a250eae3d8a46e3acedfc20420c888d2
Oct 30 07:46:16.448286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:16.448257    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:17.078426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.078329    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" event={"ID":"32661d93-ae0a-41c3-bcfc-f0d7be94433a","Type":"ContainerStarted","Data":"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"}
Oct 30 07:46:17.078426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.078393    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" event={"ID":"32661d93-ae0a-41c3-bcfc-f0d7be94433a","Type":"ContainerStarted","Data":"cd08e51a3ed8545e038bc311398b0903a250eae3d8a46e3acedfc20420c888d2"}
Oct 30 07:46:17.078426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.078421    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:17.080105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.080077    6738 generic.go:356] "Generic (PLEG): container finished" podID="534f9eef-4c59-456b-bd56-9eeac8990c24" containerID="599a00204087634dd8359fdabf8004999ba5b650c533cff4e3c9ddcdd18c3ef4" exitCode=0
Oct 30 07:46:17.080240 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.080161    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg" event={"ID":"534f9eef-4c59-456b-bd56-9eeac8990c24","Type":"ContainerDied","Data":"599a00204087634dd8359fdabf8004999ba5b650c533cff4e3c9ddcdd18c3ef4"}
Oct 30 07:46:17.141862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.141808    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" podStartSLOduration=2.14179164 podStartE2EDuration="2.14179164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:17.136436065 +0000 UTC m=+2937.569199981" watchObservedRunningTime="2026-10-30 07:46:17.14179164 +0000 UTC m=+2937.574555532"
Oct 30 07:46:17.837427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:17.837387    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:18.260086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.260060    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:18.367120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.367031    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\") pod \"534f9eef-4c59-456b-bd56-9eeac8990c24\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") "
Oct 30 07:46:18.367338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.367263    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\") pod \"534f9eef-4c59-456b-bd56-9eeac8990c24\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") "
Oct 30 07:46:18.367398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.367345    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\") pod \"534f9eef-4c59-456b-bd56-9eeac8990c24\" (UID: \"534f9eef-4c59-456b-bd56-9eeac8990c24\") "
Oct 30 07:46:18.367687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.367659    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca" pod "534f9eef-4c59-456b-bd56-9eeac8990c24" (UID: "534f9eef-4c59-456b-bd56-9eeac8990c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:18.368069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.368040    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:18.369384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.369361    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca" pod "534f9eef-4c59-456b-bd56-9eeac8990c24" (UID: "534f9eef-4c59-456b-bd56-9eeac8990c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:46:18.369560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.369539    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm" pod "534f9eef-4c59-456b-bd56-9eeac8990c24" (UID: "534f9eef-4c59-456b-bd56-9eeac8990c24"). InnerVolumeSpecName "kube-api-access-5hvcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:18.469753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.469711    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534f9eef-4c59-456b-bd56-9eeac8990c24-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:18.469753 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:18.469749    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvcm\" (UniqueName: \"kubernetes.io/projected/534f9eef-4c59-456b-bd56-9eeac8990c24-kube-api-access-5hvcm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:19.099768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:19.099731    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"
Oct 30 07:46:19.099768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:19.099748    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg" event={"ID":"534f9eef-4c59-456b-bd56-9eeac8990c24","Type":"ContainerDied","Data":"fa49839f0c82aa121664b38601aab4d70b50411d14797a5d568c95c7f9cd35a7"}
Oct 30 07:46:19.100022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:19.099812    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa49839f0c82aa121664b38601aab4d70b50411d14797a5d568c95c7f9cd35a7"
Oct 30 07:46:19.594693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:19.594654    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:21.439645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:21.439611    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:23.336405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:23.336368    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:25.308318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.307880    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:25.877635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.876695    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"]
Oct 30 07:46:25.879883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.879484    6738 cpu_manager.go:414] "containerMap: removing container" podUID="534f9eef-4c59-456b-bd56-9eeac8990c24" containerName="revoke-keys"
Oct 30 07:46:25.880109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.880091    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534f9eef-4c59-456b-bd56-9eeac8990c24" containerName="revoke-keys"
Oct 30 07:46:25.881011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.880991    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="534f9eef-4c59-456b-bd56-9eeac8990c24" containerName="revoke-keys"
Oct 30 07:46:25.887783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.887746    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:25.895638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.895610    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"]
Oct 30 07:46:25.905703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.905665    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:46:25.917736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.917699    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:46:25.980398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.979011    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:25.980398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.979114    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpnz\" (UniqueName: \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:25.980398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:25.979317    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.081414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.081372    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.081630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.081439    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpnz\" (UniqueName: \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.081630 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.081559    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.082134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.082101    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.084177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.084148    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.105667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.105632    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpnz\" (UniqueName: \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\") pod \"maas-api-revoke-sub-1a8e84d3-9t7bw\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.264663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.264577    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:26.575766 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:26.575721    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5de333b_2bfc_4b24_9e7a_dd55aa2da323.slice/crio-77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e WatchSource:0}: Error finding container 77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e: Status 404 returned error can't find the container with id 77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e
Oct 30 07:46:26.576924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:26.576898    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"]
Oct 30 07:46:27.084711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.084680    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:27.163066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.163019    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:46:27.269859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.269760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:27.296481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.296437    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw" event={"ID":"d5de333b-2bfc-4b24-9e7a-dd55aa2da323","Type":"ContainerStarted","Data":"c4aa2262ea73d63de718a58f5d911d7c29074ee3843cf3389f4ca7c109fbd431"}
Oct 30 07:46:27.296701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.296499    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw" event={"ID":"d5de333b-2bfc-4b24-9e7a-dd55aa2da323","Type":"ContainerStarted","Data":"77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e"}
Oct 30 07:46:27.377031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:27.376337    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw" podStartSLOduration=2.376316465 podStartE2EDuration="2.376316465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:27.374953041 +0000 UTC m=+2947.807716939" watchObservedRunningTime="2026-10-30 07:46:27.376316465 +0000 UTC m=+2947.809080359"
Oct 30 07:46:27.809463 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:27.809424    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e27e0c/agc-cred-4a4b01b21fc4: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:27.809463 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:27.809460    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:27.809988 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:27.809505    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials podName:2ad33057-8d5b-46ea-98ea-690880de6954 nodeName:}" failed. No retries permitted until 2026-10-30 07:46:28.309490443 +0000 UTC m=+2948.742254319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954") : secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:28.093220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:28.093134    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:28.172892 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:28.172855    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:46:28.309613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:28.309573    6738 generic.go:356] "Generic (PLEG): container finished" podID="d5de333b-2bfc-4b24-9e7a-dd55aa2da323" containerID="c4aa2262ea73d63de718a58f5d911d7c29074ee3843cf3389f4ca7c109fbd431" exitCode=0
Oct 30 07:46:28.309800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:28.309652    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw" event={"ID":"d5de333b-2bfc-4b24-9e7a-dd55aa2da323","Type":"ContainerDied","Data":"c4aa2262ea73d63de718a58f5d911d7c29074ee3843cf3389f4ca7c109fbd431"}
Oct 30 07:46:28.318442 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:28.318411    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e27e0c/agc-cred-4a4b01b21fc4: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:28.318442 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:28.318439    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:28.318679 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:28.318499    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials podName:2ad33057-8d5b-46ea-98ea-690880de6954 nodeName:}" failed. No retries permitted until 2026-10-30 07:46:29.318481609 +0000 UTC m=+2949.751245493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954") : secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:29.152513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.152476    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:29.337026 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:29.336994    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e27e0c/agc-cred-4a4b01b21fc4: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:29.337026 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:29.337022    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:29.337343 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:29.337088    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials podName:2ad33057-8d5b-46ea-98ea-690880de6954 nodeName:}" failed. No retries permitted until 2026-10-30 07:46:31.337069651 +0000 UTC m=+2951.769833539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954") : secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:29.501833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.501805    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:29.642667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.642628    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\") pod \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") "
Oct 30 07:46:29.642886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.642704    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\") pod \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") "
Oct 30 07:46:29.642886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.642850    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\" (UniqueName: \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\") pod \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\" (UID: \"d5de333b-2bfc-4b24-9e7a-dd55aa2da323\") "
Oct 30 07:46:29.643103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.643076    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca" pod "d5de333b-2bfc-4b24-9e7a-dd55aa2da323" (UID: "d5de333b-2bfc-4b24-9e7a-dd55aa2da323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:29.643692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.643665    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:29.644970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.644939    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca" pod "d5de333b-2bfc-4b24-9e7a-dd55aa2da323" (UID: "d5de333b-2bfc-4b24-9e7a-dd55aa2da323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:46:29.645088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.645031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz" pod "d5de333b-2bfc-4b24-9e7a-dd55aa2da323" (UID: "d5de333b-2bfc-4b24-9e7a-dd55aa2da323"). InnerVolumeSpecName "kube-api-access-grpnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:29.745276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.745183    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:29.745276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:29.745219    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grpnz\" (UniqueName: \"kubernetes.io/projected/d5de333b-2bfc-4b24-9e7a-dd55aa2da323-kube-api-access-grpnz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:30.091359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.091260    6738 scope.go:122] "RemoveContainer" containerID="39e92e512f8fd618738b7520aae9b3db896bdd07490cc798e12b311a9975a20e"
Oct 30 07:46:30.103980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.103953    6738 scope.go:122] "RemoveContainer" containerID="4b7b205ec0abf8cb1a865ae40a6461e9368bd555e102d35486a282f12848af3d"
Oct 30 07:46:30.115467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.115443    6738 scope.go:122] "RemoveContainer" containerID="31004a35b693c85c0b07a9283c4db88fb35b2c4a5f2265a84af24634e6182dff"
Oct 30 07:46:30.126878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.126849    6738 scope.go:122] "RemoveContainer" containerID="e0a472463c3a1416a22c7db6c35c239cab9e03f1a58cdb13888ffc187986773e"
Oct 30 07:46:30.138726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.138701    6738 scope.go:122] "RemoveContainer" containerID="5d0e1dacf46e114c0692e849c16390e0596c6c89b2e28dde5f8cfcbb6bd39514"
Oct 30 07:46:30.330268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.330237    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"
Oct 30 07:46:30.330711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.330273    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw" event={"ID":"d5de333b-2bfc-4b24-9e7a-dd55aa2da323","Type":"ContainerDied","Data":"77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e"}
Oct 30 07:46:30.330711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:30.330342    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77220a921c894f15932f7a69141b4727fc8174ba66fcf3d6480bfee7cb90199e"
Oct 30 07:46:31.003081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:31.003047    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:31.371945 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:31.371918    6738 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e27e0c/agc-cred-4a4b01b21fc4: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:31.371945 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:31.371945    6738 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc: secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:31.372406 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:31.372000    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials podName:2ad33057-8d5b-46ea-98ea-690880de6954 nodeName:}" failed. No retries permitted until 2026-10-30 07:46:35.371984791 +0000 UTC m=+2955.804748663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954") : secret "agc-cred-4a4b01b21fc4" not found
Oct 30 07:46:32.164659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.164616    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" podUID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" containerName="payload-processing" containerID="cri-o://a77ef8eaafd87a5039d3486f9611503fa328f2542e0baefae7a39e4aeec67a28" gracePeriod=25
Oct 30 07:46:32.363834 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.363787    6738 generic.go:356] "Generic (PLEG): container finished" podID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" containerID="a77ef8eaafd87a5039d3486f9611503fa328f2542e0baefae7a39e4aeec67a28" exitCode=0
Oct 30 07:46:32.364097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.364077    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" event={"ID":"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b","Type":"ContainerDied","Data":"a77ef8eaafd87a5039d3486f9611503fa328f2542e0baefae7a39e4aeec67a28"}
Oct 30 07:46:32.456737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.456707    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:46:32.488385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488345    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\") pod \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") "
Oct 30 07:46:32.488385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488389    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\") pod \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") "
Oct 30 07:46:32.488663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488578    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\") pod \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") "
Oct 30 07:46:32.488663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488609    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\") pod \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\" (UID: \"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b\") "
Oct 30 07:46:32.488784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488729    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume" pod "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" (UID: "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:32.488953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.488932    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume" pod "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" (UID: "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:32.489387 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.489370    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:32.489447 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.489389    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:32.490827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.490808    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw" pod "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" (UID: "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b"). InnerVolumeSpecName "kube-api-access-kn6mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:32.490936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.490838    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials" pod "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" (UID: "9255ad61-2a79-4b36-972c-ed2dd6eb0b1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:32.591656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.591617    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:32.591656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.591654    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6mw\" (UniqueName: \"kubernetes.io/projected/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b-kube-api-access-kn6mw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:32.860003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:32.859918    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:33.310005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.309967    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" podUID="2ad33057-8d5b-46ea-98ea-690880de6954" containerName="payload-processing" containerID="cri-o://4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86" gracePeriod=25
Oct 30 07:46:33.382795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.382763    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"
Oct 30 07:46:33.382795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.382785    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw" event={"ID":"9255ad61-2a79-4b36-972c-ed2dd6eb0b1b","Type":"ContainerDied","Data":"7da6a7dba076d679ec8a5a58d2db4eaefecc2d5b0269964326938413f104965e"}
Oct 30 07:46:33.383024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.382880    6738 scope.go:122] "RemoveContainer" containerID="a77ef8eaafd87a5039d3486f9611503fa328f2542e0baefae7a39e4aeec67a28"
Oct 30 07:46:33.441624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.441592    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:46:33.453926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.453893    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7bd6h4pkw"]
Oct 30 07:46:33.601991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.601956    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:46:33.637490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.637456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:33.696429 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.696385    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" path="/var/lib/kubelet/pods/9255ad61-2a79-4b36-972c-ed2dd6eb0b1b/volumes"
Oct 30 07:46:33.722374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.722329    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\") pod \"2ad33057-8d5b-46ea-98ea-690880de6954\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") "
Oct 30 07:46:33.722566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.722483    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\") pod \"2ad33057-8d5b-46ea-98ea-690880de6954\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") "
Oct 30 07:46:33.722645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.722631    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\") pod \"2ad33057-8d5b-46ea-98ea-690880de6954\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") "
Oct 30 07:46:33.722685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.722660    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\") pod \"2ad33057-8d5b-46ea-98ea-690880de6954\" (UID: \"2ad33057-8d5b-46ea-98ea-690880de6954\") "
Oct 30 07:46:33.723073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.723039    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume" pod "2ad33057-8d5b-46ea-98ea-690880de6954" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:33.723073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.723068    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume" pod "2ad33057-8d5b-46ea-98ea-690880de6954" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:33.723594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.723569    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:33.723716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.723597    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad33057-8d5b-46ea-98ea-690880de6954-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:33.724683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.724661    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd" pod "2ad33057-8d5b-46ea-98ea-690880de6954" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954"). InnerVolumeSpecName "kube-api-access-cxbrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:33.724787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.724712    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials" pod "2ad33057-8d5b-46ea-98ea-690880de6954" (UID: "2ad33057-8d5b-46ea-98ea-690880de6954"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:33.825474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.825382    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:33.825474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:33.825416    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbrd\" (UniqueName: \"kubernetes.io/projected/2ad33057-8d5b-46ea-98ea-690880de6954-kube-api-access-cxbrd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:34.394118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.394077    6738 generic.go:356] "Generic (PLEG): container finished" podID="2ad33057-8d5b-46ea-98ea-690880de6954" containerID="4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86" exitCode=0
Oct 30 07:46:34.394288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.394128    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"
Oct 30 07:46:34.394288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.394169    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" event={"ID":"2ad33057-8d5b-46ea-98ea-690880de6954","Type":"ContainerDied","Data":"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"}
Oct 30 07:46:34.394288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.394222    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc" event={"ID":"2ad33057-8d5b-46ea-98ea-690880de6954","Type":"ContainerDied","Data":"8344e1b5259a99b8efe1a6f040e5aaf304600b5ed05ede1f011ffacbaf04f1c3"}
Oct 30 07:46:34.394288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.394246    6738 scope.go:122] "RemoveContainer" containerID="4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"
Oct 30 07:46:34.408071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.408050    6738 scope.go:122] "RemoveContainer" containerID="4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"
Oct 30 07:46:34.408441 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:34.408413    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86\": container with ID starting with 4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86 not found: ID does not exist" containerID="4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"
Oct 30 07:46:34.408564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.408446    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86"} err="failed to get container status \"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86\": rpc error: code = NotFound desc = could not find container \"4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86\": container with ID starting with 4041b8e60f6d68a8aba550efccc126a36587a04edbda0666345807d61e651e86 not found: ID does not exist"
Oct 30 07:46:34.450488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.450447    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:46:34.461494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.461464    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-84797lnpc"]
Oct 30 07:46:34.756132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:34.756040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:35.692488 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:35.692448    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad33057-8d5b-46ea-98ea-690880de6954" path="/var/lib/kubelet/pods/2ad33057-8d5b-46ea-98ea-690880de6954/volumes"
Oct 30 07:46:36.348458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:36.348391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:36.583684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:36.583643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:36.798535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:36.798496    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:37.521094 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:37.521057    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:46:37.827783 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:37.827696    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:37.914048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:37.914009    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:38.054251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:38.054217    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:38.567902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:38.567859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:38.765394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:38.765360    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:38.848895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:38.848809    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:40.736499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:40.736459    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:42.598881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:42.598842    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:42.828542 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:42.828482    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" podUID="d610984a-46e3-4855-b8a8-76863d2be9ff" containerName="payload-processing" containerID="cri-o://d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b" gracePeriod=25
Oct 30 07:46:43.033893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.033850    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"]
Oct 30 07:46:43.044343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.044303    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba310e5-x9tfw"]
Oct 30 07:46:43.129238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.129212    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:43.271545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.271510    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\") pod \"d610984a-46e3-4855-b8a8-76863d2be9ff\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") "
Oct 30 07:46:43.271740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.271564    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\") pod \"d610984a-46e3-4855-b8a8-76863d2be9ff\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") "
Oct 30 07:46:43.271740 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.271607    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\") pod \"d610984a-46e3-4855-b8a8-76863d2be9ff\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") "
Oct 30 07:46:43.271896 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.271878    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\") pod \"d610984a-46e3-4855-b8a8-76863d2be9ff\" (UID: \"d610984a-46e3-4855-b8a8-76863d2be9ff\") "
Oct 30 07:46:43.271977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.271957    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume" pod "d610984a-46e3-4855-b8a8-76863d2be9ff" (UID: "d610984a-46e3-4855-b8a8-76863d2be9ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:43.272041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.272014    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume" pod "d610984a-46e3-4855-b8a8-76863d2be9ff" (UID: "d610984a-46e3-4855-b8a8-76863d2be9ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:43.272698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.272672    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:43.272698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.272695    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d610984a-46e3-4855-b8a8-76863d2be9ff-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:43.273809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.273784    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25" pod "d610984a-46e3-4855-b8a8-76863d2be9ff" (UID: "d610984a-46e3-4855-b8a8-76863d2be9ff"). InnerVolumeSpecName "kube-api-access-fdk25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:43.273907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.273892    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials" pod "d610984a-46e3-4855-b8a8-76863d2be9ff" (UID: "d610984a-46e3-4855-b8a8-76863d2be9ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:43.374781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.374747    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdk25\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-kube-api-access-fdk25\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:43.374781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.374781    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d610984a-46e3-4855-b8a8-76863d2be9ff-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:43.508214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.508130    6738 generic.go:356] "Generic (PLEG): container finished" podID="d610984a-46e3-4855-b8a8-76863d2be9ff" containerID="d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b" exitCode=0
Oct 30 07:46:43.508214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.508178    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"
Oct 30 07:46:43.508491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.508210    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" event={"ID":"d610984a-46e3-4855-b8a8-76863d2be9ff","Type":"ContainerDied","Data":"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"}
Oct 30 07:46:43.508491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.508278    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6" event={"ID":"d610984a-46e3-4855-b8a8-76863d2be9ff","Type":"ContainerDied","Data":"e663ed30beb41ed5cbb34da42c027a7503f1ab5eed17e62ef64aa5a9ccecaea7"}
Oct 30 07:46:43.508491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.508355    6738 scope.go:122] "RemoveContainer" containerID="d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"
Oct 30 07:46:43.520693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.520676    6738 scope.go:122] "RemoveContainer" containerID="d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"
Oct 30 07:46:43.521011 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:43.520991    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b\": container with ID starting with d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b not found: ID does not exist" containerID="d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"
Oct 30 07:46:43.521128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.521024    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b"} err="failed to get container status \"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b\": rpc error: code = NotFound desc = could not find container \"d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b\": container with ID starting with d368973f7874d6ead483cb83ed988b59f984a588ce6bbe8f99cfdd788eeff99b not found: ID does not exist"
Oct 30 07:46:43.565715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.565684    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:43.576528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.576493    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8d0c2f/payload-processing-external-model-e2e-extmod-a-8d0c2f-7b88k4kq6"]
Oct 30 07:46:43.692563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.692526    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09392c7f-169f-4922-9e68-aa5284dd4963" path="/var/lib/kubelet/pods/09392c7f-169f-4922-9e68-aa5284dd4963/volumes"
Oct 30 07:46:43.693131 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.693115    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d610984a-46e3-4855-b8a8-76863d2be9ff" path="/var/lib/kubelet/pods/d610984a-46e3-4855-b8a8-76863d2be9ff/volumes"
Oct 30 07:46:43.765880 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.765769    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" podUID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" containerName="payload-processing" containerID="cri-o://2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44" gracePeriod=25
Oct 30 07:46:43.930053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:43.930021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:44.081243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.081211    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:44.195626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.195586    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\") pod \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") "
Oct 30 07:46:44.195821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.195734    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\") pod \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") "
Oct 30 07:46:44.195821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.195793    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\") pod \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") "
Oct 30 07:46:44.196045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.196024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\") pod \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\" (UID: \"32661d93-ae0a-41c3-bcfc-f0d7be94433a\") "
Oct 30 07:46:44.196045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.196028    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume" pod "32661d93-ae0a-41c3-bcfc-f0d7be94433a" (UID: "32661d93-ae0a-41c3-bcfc-f0d7be94433a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:44.196453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.196428    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume" pod "32661d93-ae0a-41c3-bcfc-f0d7be94433a" (UID: "32661d93-ae0a-41c3-bcfc-f0d7be94433a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:44.196628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.196610    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:44.196706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.196629    6738 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32661d93-ae0a-41c3-bcfc-f0d7be94433a-routing-overlay-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:44.197971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.197948    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials" pod "32661d93-ae0a-41c3-bcfc-f0d7be94433a" (UID: "32661d93-ae0a-41c3-bcfc-f0d7be94433a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:44.198197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.198176    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r" pod "32661d93-ae0a-41c3-bcfc-f0d7be94433a" (UID: "32661d93-ae0a-41c3-bcfc-f0d7be94433a"). InnerVolumeSpecName "kube-api-access-g8d7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:44.298101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.298000    6738 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-provider-credentials\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:44.298101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.298036    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8d7r\" (UniqueName: \"kubernetes.io/projected/32661d93-ae0a-41c3-bcfc-f0d7be94433a-kube-api-access-g8d7r\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:46:44.410786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.410743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:44.518719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.518679    6738 generic.go:356] "Generic (PLEG): container finished" podID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" containerID="2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44" exitCode=0
Oct 30 07:46:44.518894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.518732    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"
Oct 30 07:46:44.518894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.518743    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" event={"ID":"32661d93-ae0a-41c3-bcfc-f0d7be94433a","Type":"ContainerDied","Data":"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"}
Oct 30 07:46:44.518894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.518826    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw" event={"ID":"32661d93-ae0a-41c3-bcfc-f0d7be94433a","Type":"ContainerDied","Data":"cd08e51a3ed8545e038bc311398b0903a250eae3d8a46e3acedfc20420c888d2"}
Oct 30 07:46:44.518894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.518852    6738 scope.go:122] "RemoveContainer" containerID="2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"
Oct 30 07:46:44.535420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.535384    6738 scope.go:122] "RemoveContainer" containerID="2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"
Oct 30 07:46:44.535816 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:46:44.535786    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44\": container with ID starting with 2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44 not found: ID does not exist" containerID="2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"
Oct 30 07:46:44.535924 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.535828    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44"} err="failed to get container status \"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44\": rpc error: code = NotFound desc = could not find container \"2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44\": container with ID starting with 2e3e78e479e21bd47fadd031e89bfdeca0f95ec26db499f00ef88303cfa65d44 not found: ID does not exist"
Oct 30 07:46:44.576109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.576026    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:44.582024 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.581995    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cstxlw"]
Oct 30 07:46:44.645891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.645859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:44.942879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.942842    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:46:44.944163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944132    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d610984a-46e3-4855-b8a8-76863d2be9ff" containerName="payload-processing"
Oct 30 07:46:44.944163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944160    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610984a-46e3-4855-b8a8-76863d2be9ff" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944188    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2ad33057-8d5b-46ea-98ea-690880de6954" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944197    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad33057-8d5b-46ea-98ea-690880de6954" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944227    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944236    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944271    6738 cpu_manager.go:414] "containerMap: removing container" podUID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944277    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" containerName="payload-processing"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944337    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d5de333b-2bfc-4b24-9e7a-dd55aa2da323" containerName="revoke-keys"
Oct 30 07:46:44.944404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944344    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5de333b-2bfc-4b24-9e7a-dd55aa2da323" containerName="revoke-keys"
Oct 30 07:46:44.944843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944561    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" containerName="payload-processing"
Oct 30 07:46:44.944843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944586    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5de333b-2bfc-4b24-9e7a-dd55aa2da323" containerName="revoke-keys"
Oct 30 07:46:44.944843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944604    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610984a-46e3-4855-b8a8-76863d2be9ff" containerName="payload-processing"
Oct 30 07:46:44.944843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944617    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad33057-8d5b-46ea-98ea-690880de6954" containerName="payload-processing"
Oct 30 07:46:44.944843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.944628    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9255ad61-2a79-4b36-972c-ed2dd6eb0b1b" containerName="payload-processing"
Oct 30 07:46:44.950709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.950689    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:44.965425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.965380    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:46:44.968868 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.968842    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8c0e05-openshift-default-dockercfg-dmjmv\""
Oct 30 07:46:44.992116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:44.992083    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118468    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118617    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118652    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118678    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118706    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118758    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118796    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.118897    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfgs\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.122321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.119013    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222219    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222421    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222550    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222574    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222624    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222667    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222700    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.222787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfgs\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.223971    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.224582    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.225169    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.225389    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.225602    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.235368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.233284    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.255316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.248863    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.255316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.250593    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfgs\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.260321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.257877    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\") pod \"e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") " pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.273315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.271574    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:45.471776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.471738    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:45.607385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.606274    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:45.652303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.652258    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:46:45.659053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.659007    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:45.659181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.659111    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:45.659181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.659156    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:45.693252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.693221    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32661d93-ae0a-41c3-bcfc-f0d7be94433a" path="/var/lib/kubelet/pods/32661d93-ae0a-41c3-bcfc-f0d7be94433a/volumes"
Oct 30 07:46:45.859452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:45.859345    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:46.083618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.083586    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:46.569317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.568624    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerStarted","Data":"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"}
Oct 30 07:46:46.569317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.568713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerStarted","Data":"6fc5b2a31fe9b5f13ec15cc8d8557c590a362efbd5aac1b0c68a0cbcf0c4a3d7"}
Oct 30 07:46:46.654278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.654214    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podStartSLOduration=2.654192945 podStartE2EDuration="2.654192945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:46.647093041 +0000 UTC m=+2967.079856934" watchObservedRunningTime="2026-10-30 07:46:46.654192945 +0000 UTC m=+2967.086956838"
Oct 30 07:46:46.687080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.687048    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:46:46.692380 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.692350    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.744313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.730931    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0e9723-openshift-default-dockercfg-bnplm\""
Oct 30 07:46:46.768951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.768065    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769350 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769325    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769567 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769549    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769679    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769826    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769866    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwrb\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.769979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.769944    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.770045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.770015    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.770157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.770138    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.783273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.783238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:46.785392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.785360    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887559    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887638    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887693    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887742    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887770    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwrb\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887832    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.887976    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.894604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.888170    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.895600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.895442    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.907321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.896093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.907321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.901188    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.907321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.901543    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.907321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.901793    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.916860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.916147    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.916860 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.916725    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:46.922862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.920648    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.927932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.927894    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.946280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.940982    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwrb\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:46.952759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:46.952666    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\") pod \"e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") " pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:47.092319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.090626    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:47.298347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.288823    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:47.298347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.290804    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:47.332629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.329471    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:47.431485 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:47.431446    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10021836_b642_4e73_9574_83d9b4f654a3.slice/crio-b109089334c79460470818c5bb8b682b7432a902b3638ec21551fec3a92eb662 WatchSource:0}: Error finding container b109089334c79460470818c5bb8b682b7432a902b3638ec21551fec3a92eb662: Status 404 returned error can't find the container with id b109089334c79460470818c5bb8b682b7432a902b3638ec21551fec3a92eb662
Oct 30 07:46:47.432511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.432486    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:46:47.434964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.434926    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:47.435041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.435001    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:47.435041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.435035    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:47.601830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.601786    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerStarted","Data":"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"}
Oct 30 07:46:47.603982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.601861    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerStarted","Data":"b109089334c79460470818c5bb8b682b7432a902b3638ec21551fec3a92eb662"}
Oct 30 07:46:47.603982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.602213    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:47.605127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.605104    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:46:47.652234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:47.652175    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podStartSLOduration=1.6521602770000001 podStartE2EDuration="1.652160277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:47.648989506 +0000 UTC m=+2968.081753398" watchObservedRunningTime="2026-10-30 07:46:47.652160277 +0000 UTC m=+2968.084924169"
Oct 30 07:46:48.094420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:48.094387    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:48.095782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:48.095578    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:46:48.095782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:48.095635    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:46:48.237438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:48.237398    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:49.098171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.098135    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:49.349385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.346413    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:49.494766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.494699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:49.618735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.618646    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:49.619833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.619808    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:46:49.987318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:49.986859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:50.227322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:50.222985    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:50.366595 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:50.366511    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:50.536457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:50.536422    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:51.719855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:51.719816    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:52.194993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:52.194954    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:52.329581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:52.329544    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:53.269923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.269835    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:53.403811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.403769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:53.446532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.446492    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:46:53.453080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.453049    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.474077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.474040    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:46:53.480641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.480611    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0e9723-dockercfg-4m8sb\""
Oct 30 07:46:53.578379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.574775    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0e9723\""
Oct 30 07:46:53.682476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.682429    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.698975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.698940    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdgw\" (UniqueName: \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.806800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.806763    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.816200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.816152    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdgw\" (UniqueName: \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.818018 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.808004    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:53.871558 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:53.862044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdgw\" (UniqueName: \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\") pod \"payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:54.072493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.072449    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:54.237417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.237376    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:54.407102 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:46:54.407072    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca08662_4a18_48b5_9ab3_a203f79f8f29.slice/crio-de84ac9da89c14bbc3df35a7a44ad8fcea89550c7b505ddbd8752957f15ae337 WatchSource:0}: Error finding container de84ac9da89c14bbc3df35a7a44ad8fcea89550c7b505ddbd8752957f15ae337: Status 404 returned error can't find the container with id de84ac9da89c14bbc3df35a7a44ad8fcea89550c7b505ddbd8752957f15ae337
Oct 30 07:46:54.409547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.409519    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:46:54.806425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.806326    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" event={"ID":"cca08662-4a18-48b5-9ab3-a203f79f8f29","Type":"ContainerStarted","Data":"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"}
Oct 30 07:46:54.806425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.806398    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:46:54.806425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.806422    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" event={"ID":"cca08662-4a18-48b5-9ab3-a203f79f8f29","Type":"ContainerStarted","Data":"de84ac9da89c14bbc3df35a7a44ad8fcea89550c7b505ddbd8752957f15ae337"}
Oct 30 07:46:54.852807 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:54.852742    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" podStartSLOduration=1.852727617 podStartE2EDuration="1.852727617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:54.847755411 +0000 UTC m=+2975.280519305" watchObservedRunningTime="2026-10-30 07:46:54.852727617 +0000 UTC m=+2975.285491511"
Oct 30 07:46:56.051666 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:56.051617    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:57.836282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.836194    6738 generic.go:356] "Generic (PLEG): container finished" podID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerID="94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941" exitCode=0
Oct 30 07:46:57.836282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.836256    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerDied","Data":"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"}
Oct 30 07:46:57.836812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.836664    6738 scope.go:122] "RemoveContainer" containerID="94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"
Oct 30 07:46:57.838769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.838734    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:57.838891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.838824    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:57.838891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.838874    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:46:57.956798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:57.956758    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:46:58.418405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:58.418371    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:46:58.858281 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:58.858180    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerStarted","Data":"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"}
Oct 30 07:46:59.019402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.019283    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:46:59.020579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.020545    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:46:59.020739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.020597    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:46:59.910099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.910060    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:46:59.914473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.914450    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:46:59.933667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.933634    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:46:59.941600 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.941570    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ccac6f-serving-cert\""
Oct 30 07:46:59.996319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.992477    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sjx\" (UniqueName: \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:46:59.999637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:46:59.999595    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.000194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.000166    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.000931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.000907    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.010118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.010083    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:00.019101 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.019071    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:47:00.019274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.019120    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.112768    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sjx\" (UniqueName: \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.112983    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.113201    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.113242    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.114243    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:00.114670    6738 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-serving-cert: secret "maas-api-e2e-inf-a-ccac6f-serving-cert" not found
Oct 30 07:47:00.116318 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:00.114733    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls podName:e756f6be-9b0f-4a2b-b361-3ee56382a7cf nodeName:}" failed. No retries permitted until 2026-10-30 07:47:00.614714005 +0000 UTC m=+2981.047477890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls") pod "maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" (UID: "e756f6be-9b0f-4a2b-b361-3ee56382a7cf") : secret "maas-api-e2e-inf-a-ccac6f-serving-cert" not found
Oct 30 07:47:00.136647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.136609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:00.157390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.142421    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sjx\" (UniqueName: \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.165419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.158040    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.439010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.438915    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:00.621832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.621789    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.624925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.624889    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:00.834225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:00.834140    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:01.027465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:01.026997    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:47:01.027465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:01.027058    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:47:01.283598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:01.283553    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:47:01.326318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:01.321574    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:47:01.959967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:01.959930    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" event={"ID":"e756f6be-9b0f-4a2b-b361-3ee56382a7cf","Type":"ContainerStarted","Data":"ca75cb01f1aba505ca53b7137c06aa5806e9376a15cbcc565e10ad4ec32f089c"}
Oct 30 07:47:02.019920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:02.019874    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:47:02.020125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:02.019939    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:47:02.359789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:02.359754    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:02.994725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:02.994676    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" event={"ID":"e756f6be-9b0f-4a2b-b361-3ee56382a7cf","Type":"ContainerStarted","Data":"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"}
Oct 30 07:47:02.995859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:02.995837    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:03.020021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:03.019987    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:47:03.020207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:03.020039    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:47:03.039975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:03.039922    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" podStartSLOduration=3.53254368 podStartE2EDuration="4.039905808s" totalImagesPullingTime="507.362128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:03.037574232 +0000 UTC m=+2983.470338128" watchObservedRunningTime="2026-10-30 07:47:03.039905808 +0000 UTC m=+2983.472669735"
Oct 30 07:47:04.019670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:04.019634    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Oct 30 07:47:04.020110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:04.019691    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Oct 30 07:47:04.272273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:04.272180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:05.024445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:05.024416    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:47:05.024959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:05.024812    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:47:05.025517 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:05.025493    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:47:05.821955 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:05.821924    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:47:06.133857 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:06.133821    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:07.520455 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:07.520416    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:47:08.049768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:08.049735    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:09.879016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:09.878982    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:10.023689 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:10.023661    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:47:11.787211 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:11.787176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:13.643104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:13.643043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:14.484149 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:14.484113    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:15.446420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:15.446383    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:17.547317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:17.543895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:19.944166 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:19.944128    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:21.893417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:21.893377    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:23.794016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:23.793979    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:25.668467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:25.668427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:27.509308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:27.509256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:28.436424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:28.434992    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": read tcp 10.133.0.2:51614->10.133.1.139:15021: read: connection reset by peer" start-of-body=
Oct 30 07:47:28.436424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:28.435075    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": read tcp 10.133.0.2:51614->10.133.1.139:15021: read: connection reset by peer"
Oct 30 07:47:29.302716 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.302675    6738 generic.go:356] "Generic (PLEG): container finished" podID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerID="4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d" exitCode=0
Oct 30 07:47:29.303247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.302755    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerDied","Data":"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"}
Oct 30 07:47:29.303400 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.303381    6738 scope.go:122] "RemoveContainer" containerID="4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"
Oct 30 07:47:29.305872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.305836    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:47:29.306028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.305923    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:47:29.306028 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.305960    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:47:29.310249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:29.310220    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:30.314965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:30.314931    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerStarted","Data":"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"}
Oct 30 07:47:31.059066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:31.059030    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:47:31.059269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:31.059227    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body=
Oct 30 07:47:31.059351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:31.059322    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused"
Oct 30 07:47:31.204608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:31.204570    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:32.034964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:32.034925    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"]
Oct 30 07:47:32.047061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:32.047024    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7dfd28-nxnsb"]
Oct 30 07:47:32.059446 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:32.059414    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body=
Oct 30 07:47:32.059627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:32.059467    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused"
Oct 30 07:47:33.059230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:33.059191    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body=
Oct 30 07:47:33.059827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:33.059265    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused"
Oct 30 07:47:33.092946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:33.092908    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:33.703702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:33.703666    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97afbe8-0256-4559-89b3-7e291813240c" path="/var/lib/kubelet/pods/a97afbe8-0256-4559-89b3-7e291813240c/volumes"
Oct 30 07:47:33.755599 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:33.755557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:34.059242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:34.059152    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body=
Oct 30 07:47:34.059242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:34.059226    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused"
Oct 30 07:47:35.000878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:35.000841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:35.060636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:35.060599    6738 patch_prober.go:29] interesting pod/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body=
Oct 30 07:47:35.061382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:35.061346    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused"
Oct 30 07:47:36.063812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:36.063777    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:47:36.064343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:36.064289    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:47:36.065057 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:36.065024    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:47:36.856197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:36.856159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:37.521970 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:37.521930    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:47:38.688938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:38.688888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:40.519508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:40.519475    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:42.392058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:42.392020    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:44.139443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:44.139404    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:45.944608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:45.944572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:47.745530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:47.745495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:49.720657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:49.720620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:51.464129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.464090    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:47:51.468433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.468411    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.488377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.488342    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:47:51.495711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.495682    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8c0e05-serving-cert\""
Oct 30 07:47:51.548421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.548381    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.548612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.548523    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.548717 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.548693    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.548815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.548801    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7g2\" (UniqueName: \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.556519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.556490    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:51.667568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.667522    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.667956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.667931    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.668283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.668255    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.672516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.672462    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7g2\" (UniqueName: \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.680970 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:51.680938    6738 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-serving-cert: secret "maas-api-e2e-inf-b-8c0e05-serving-cert" not found
Oct 30 07:47:51.681511 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:47:51.681490    6738 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls podName:0c9ee0f5-d825-422f-b9ab-13404a17ce66 nodeName:}" failed. No retries permitted until 2026-10-30 07:47:52.181465698 +0000 UTC m=+3032.614229606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls") pod "maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" (UID: "0c9ee0f5-d825-422f-b9ab-13404a17ce66") : secret "maas-api-e2e-inf-b-8c0e05-serving-cert" not found
Oct 30 07:47:51.690954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.687950    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.703585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.703545    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:51.727545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.712932    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.728343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.728271    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7g2\" (UniqueName: \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:51.982107 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:51.982008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:52.185827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:52.185786    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:52.188389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:52.188358    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c0e05-84598d854-ntj54\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:52.383286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:52.383254    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:52.716603 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:47:52.716566    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9ee0f5_d825_422f_b9ab_13404a17ce66.slice/crio-4b152feb11c01758c10965c0306714879668e94fb751cf4e1f7f9ba6f65c6d90 WatchSource:0}: Error finding container 4b152feb11c01758c10965c0306714879668e94fb751cf4e1f7f9ba6f65c6d90: Status 404 returned error can't find the container with id 4b152feb11c01758c10965c0306714879668e94fb751cf4e1f7f9ba6f65c6d90
Oct 30 07:47:52.719385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:52.719358    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:47:53.527455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.527413    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" event={"ID":"0c9ee0f5-d825-422f-b9ab-13404a17ce66","Type":"ContainerStarted","Data":"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"}
Oct 30 07:47:53.527659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.527485    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:47:53.527659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.527509    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" event={"ID":"0c9ee0f5-d825-422f-b9ab-13404a17ce66","Type":"ContainerStarted","Data":"4b152feb11c01758c10965c0306714879668e94fb751cf4e1f7f9ba6f65c6d90"}
Oct 30 07:47:53.573420 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.573364    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" podStartSLOduration=2.208349998 podStartE2EDuration="2.573349212s" totalImagesPullingTime="364.999214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:53.570271108 +0000 UTC m=+3034.003035005" watchObservedRunningTime="2026-10-30 07:47:53.573349212 +0000 UTC m=+3034.006113105"
Oct 30 07:47:53.679157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.679112    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:53.801549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.801458    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:53.919932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:53.919886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:55.642234 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:55.642191    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:57.515841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:57.515807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:59.547791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:59.547746    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:47:59.549401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:47:59.549379    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:48:01.545223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:01.545185    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:03.397962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:03.397923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:05.229422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:05.229383    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:07.440579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:07.440536    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:07.557797 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:48:07.557750    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:48:09.466030 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:09.465998    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:11.333302 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:11.333250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:12.665718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:12.665682    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:13.129596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:13.129535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:13.240172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:13.240134    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:13.677343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:13.677287    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:13.917997 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:13.917961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:15.220009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:15.219967    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:17.208109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:17.208074    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:17.820270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:17.820230    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:17.950511 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:17.950475    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:18.115761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:18.115658    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:19.298508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:19.298462    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:19.621508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:19.621483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:19.745346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:19.745310    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:21.320916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:21.320882    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:23.156886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:23.156843    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:25.013887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:25.013848    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:27.119357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:27.111923    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:29.297692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:29.297473    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:30.265251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:30.265219    6738 scope.go:122] "RemoveContainer" containerID="8c05f6efb324eeb6cee4f4c4934334140f58a1b996f9ae68c7bca8319bc780b5"
Oct 30 07:48:30.327802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:30.327776    6738 scope.go:122] "RemoveContainer" containerID="02a2f2aa1c1b9e02b8db10d6c4d9efa42c98645f67120d441008bbb386635ea6"
Oct 30 07:48:31.346334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:31.346285    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:33.236873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:33.236815    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:33.643541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:33.643491    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:48:33.643777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:33.643565    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:48:33.904641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:33.904544    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:35.331949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:35.331912    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:35.661505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:35.661462    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:48:35.661703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:35.661530    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:48:37.141133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.141096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:37.285461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.285429    6738 generic.go:356] "Generic (PLEG): container finished" podID="10021836-b642-4e73-9574-83d9b4f654a3" containerID="a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443" exitCode=0
Oct 30 07:48:37.285651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.285504    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerDied","Data":"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"}
Oct 30 07:48:37.286023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.286005    6738 scope.go:122] "RemoveContainer" containerID="a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"
Oct 30 07:48:37.287762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.287735    6738 generic.go:356] "Generic (PLEG): container finished" podID="023aac0b-5913-4781-b327-909efbb5fe24" containerID="1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2" exitCode=0
Oct 30 07:48:37.287881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.287814    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerDied","Data":"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"}
Oct 30 07:48:37.288326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.288309    6738 scope.go:122] "RemoveContainer" containerID="1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"
Oct 30 07:48:37.288701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.288670    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.288790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.288771    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.288843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.288826    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.290486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.290457    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.290574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.290536    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.290615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:37.290580    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:48:37.549316 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:48:37.548957    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:48:38.323908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:38.323866    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerStarted","Data":"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"}
Oct 30 07:48:38.326651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:38.326616    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerStarted","Data":"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"}
Oct 30 07:48:39.052283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.052245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:39.091190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.091159    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:48:39.091843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.091814    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:48:39.092051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.092026    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:48:39.275745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.275705    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:48:39.276150 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.276052    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 07:48:39.276338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:39.276314    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 07:48:40.095320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:40.094681    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:48:40.095320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:40.094765    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:48:40.276163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:40.276123    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 07:48:40.276470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:40.276435    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 07:48:41.093322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:41.092550    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:41.093322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:41.093156    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:48:41.093322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:41.093224    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:48:41.289648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:41.289611    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 07:48:41.290252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:41.290220    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 07:48:42.093767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:42.093724    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:48:42.094091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:42.094058    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:48:42.283320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:42.280759    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 07:48:42.283320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:42.280833    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 07:48:43.097950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:43.097908    6738 patch_prober.go:29] interesting pod/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused" start-of-body=
Oct 30 07:48:43.098618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:43.098586    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.147:15021/healthz/ready\": dial tcp 10.133.1.147:15021: connect: connection refused"
Oct 30 07:48:43.218455 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:43.218418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:43.273741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:43.273702    6738 patch_prober.go:29] interesting pod/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 07:48:43.274033 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:43.274002    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 07:48:44.096844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.096811    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:48:44.097354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.097325    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:48:44.098111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.098088    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:48:44.277991 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.277957    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:48:44.278556 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.278540    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:48:44.279433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:44.279407    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:48:45.225375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:45.225334    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:47.172547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:47.172515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:48.846622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:48.846586    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:48.981804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:48.981769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:50.856958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:50.856905    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:52.772137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:52.772098    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:54.539827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:54.539783    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:56.424286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:56.424255    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:48:58.255385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:48:58.255346    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:00.312947 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:00.312910    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:02.380128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:02.380081    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:03.117984 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:03.117944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:03.985583 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:03.985546    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:04.627889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:04.627844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:04.744257 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:04.744214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:04.874894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:04.874858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:05.078092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:05.077987    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:05.682504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:05.681998    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:06.416413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:06.416370    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:07.521704 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:49:07.521669    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:49:08.345607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:08.345566    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:09.025987 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.025948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:09.037210 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.037178    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"]
Oct 30 07:49:09.048729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.048698    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261f509c-v8bxm"]
Oct 30 07:49:09.156402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.156363    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:09.269041 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.269000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:09.697199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:09.697158    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ef30a5-18db-4bed-95a6-963facbcd2ea" path="/var/lib/kubelet/pods/76ef30a5-18db-4bed-95a6-963facbcd2ea/volumes"
Oct 30 07:49:10.466428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:10.466389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:10.873929 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:10.873895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:10.997767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:10.997700    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:11.298406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.288612    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:49:11.448319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.447592    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.507366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.503534    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8514a3-dockercfg-hbvq7\""
Oct 30 07:49:11.508611 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.508575    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:49:11.521128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.521099    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8514a3\""
Oct 30 07:49:11.598229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.598131    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.598411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.598378    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx966\" (UniqueName: \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.700493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.700443    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.700706 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.700680    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx966\" (UniqueName: \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.701684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.701657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.725130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.725094    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx966\" (UniqueName: \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\") pod \"payload-processing-e2e-inf-b-8514a3-89fd46658-r569c\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:11.839390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:11.833733    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:12.657067 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:49:12.657036    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bd152f_53b5_436b_b75e_954f4a3c6429.slice/crio-fef4c2bdd60f5a86c3abeea5e21f5058bbefe52e3f8b7e1dd150188d987dc733 WatchSource:0}: Error finding container fef4c2bdd60f5a86c3abeea5e21f5058bbefe52e3f8b7e1dd150188d987dc733: Status 404 returned error can't find the container with id fef4c2bdd60f5a86c3abeea5e21f5058bbefe52e3f8b7e1dd150188d987dc733
Oct 30 07:49:12.657521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:12.657466    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:49:12.736196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:12.736154    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" event={"ID":"75bd152f-53b5-436b-b75e-954f4a3c6429","Type":"ContainerStarted","Data":"fef4c2bdd60f5a86c3abeea5e21f5058bbefe52e3f8b7e1dd150188d987dc733"}
Oct 30 07:49:12.933372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:12.933273    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:13.746927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:13.746887    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" event={"ID":"75bd152f-53b5-436b-b75e-954f4a3c6429","Type":"ContainerStarted","Data":"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"}
Oct 30 07:49:13.747467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:13.746965    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:13.792108 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:13.792050    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" podStartSLOduration=2.792035319 podStartE2EDuration="2.792035319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:13.78782473 +0000 UTC m=+3114.220588613" watchObservedRunningTime="2026-10-30 07:49:13.792035319 +0000 UTC m=+3114.224799230"
Oct 30 07:49:14.925734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:14.925699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:16.861845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:16.861808    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:18.838140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:18.838091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:19.031886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:19.031844    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"]
Oct 30 07:49:19.043969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:19.043925    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faac18c2-ddlh2"]
Oct 30 07:49:19.694569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:19.694537    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6484ff4-e6fd-4f37-883c-0d0a8d3e391e" path="/var/lib/kubelet/pods/f6484ff4-e6fd-4f37-883c-0d0a8d3e391e/volumes"
Oct 30 07:49:20.627031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:20.626985    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:22.498501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:22.498460    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:24.306759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:24.306721    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:24.428453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:24.428412    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:24.759731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:24.759701    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:49:26.380844 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:26.380792    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:28.348386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:28.348344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:30.102284 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:30.102251    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:30.840437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:30.840399    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:31.944048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:31.943997    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:33.738869 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:33.738815    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:34.049435 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:34.049340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:35.541845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:35.541802    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:36.322726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:36.322693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:37.452353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:37.452313    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:37.580637 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:49:37.580594    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:49:39.350158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:39.350123    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:41.473827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:41.473769    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:43.262668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.262630    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"]
Oct 30 07:49:43.267464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.267438    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.284939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.284869    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:49:43.285395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.285361    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"]
Oct 30 07:49:43.291484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.291454    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:49:43.338065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.338030    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:43.427754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.427703    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtp5\" (UniqueName: \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.427754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.427754    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.428050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.427859    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.529437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.529341    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.529767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.529746    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtp5\" (UniqueName: \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.529824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.529777    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.530546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.530524    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.531908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.531885    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.553206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.553175    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtp5\" (UniqueName: \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\") pod \"maas-api-revoke-sub-013a4094-jt4hk\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.583887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.583852    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:43.893171 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:49:43.893130    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8281e1_8447_42a0_9261_f96c52c71e6d.slice/crio-f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa WatchSource:0}: Error finding container f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa: Status 404 returned error can't find the container with id f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa
Oct 30 07:49:43.893649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:43.893542    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"]
Oct 30 07:49:44.051675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:44.051640    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk" event={"ID":"2c8281e1-8447-42a0-9261-f96c52c71e6d","Type":"ContainerStarted","Data":"f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa"}
Oct 30 07:49:45.061693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:45.061604    6738 generic.go:356] "Generic (PLEG): container finished" podID="2c8281e1-8447-42a0-9261-f96c52c71e6d" containerID="fd571c331833444869ed84059c3abf5d089584c18f4711bb10b13453a0099d51" exitCode=0
Oct 30 07:49:45.062104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:45.061691    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk" event={"ID":"2c8281e1-8447-42a0-9261-f96c52c71e6d","Type":"ContainerDied","Data":"fd571c331833444869ed84059c3abf5d089584c18f4711bb10b13453a0099d51"}
Oct 30 07:49:45.150513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:45.150478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:46.255177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.255149    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:46.272671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.272639    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\") pod \"2c8281e1-8447-42a0-9261-f96c52c71e6d\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") "
Oct 30 07:49:46.272856 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.272698    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\" (UniqueName: \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\") pod \"2c8281e1-8447-42a0-9261-f96c52c71e6d\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") "
Oct 30 07:49:46.272926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.272908    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\") pod \"2c8281e1-8447-42a0-9261-f96c52c71e6d\" (UID: \"2c8281e1-8447-42a0-9261-f96c52c71e6d\") "
Oct 30 07:49:46.273091 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.273067    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca" pod "2c8281e1-8447-42a0-9261-f96c52c71e6d" (UID: "2c8281e1-8447-42a0-9261-f96c52c71e6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:49:46.273683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.273663    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:49:46.274931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.274903    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5" pod "2c8281e1-8447-42a0-9261-f96c52c71e6d" (UID: "2c8281e1-8447-42a0-9261-f96c52c71e6d"). InnerVolumeSpecName "kube-api-access-xhtp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:49:46.274931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.274911    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca" pod "2c8281e1-8447-42a0-9261-f96c52c71e6d" (UID: "2c8281e1-8447-42a0-9261-f96c52c71e6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:49:46.374936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.374891    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtp5\" (UniqueName: \"kubernetes.io/projected/2c8281e1-8447-42a0-9261-f96c52c71e6d-kube-api-access-xhtp5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:49:46.374936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.374931    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8281e1-8447-42a0-9261-f96c52c71e6d-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:49:46.991652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:46.991607    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:47.090884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:47.090852    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"
Oct 30 07:49:47.091174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:47.090874    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk" event={"ID":"2c8281e1-8447-42a0-9261-f96c52c71e6d","Type":"ContainerDied","Data":"f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa"}
Oct 30 07:49:47.091366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:47.091201    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6982a6db34229538e6e22df0b7ba10e357a7bad89be18266bf49eb25b65beaa"
Oct 30 07:49:47.634891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:47.634848    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:48.812819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:48.812778    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:50.640066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:50.640026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:52.467415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:52.467382    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:54.322078 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:54.322041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:54.435811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:54.435739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:55.496009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.495968    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"]
Oct 30 07:49:55.497022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.497002    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2c8281e1-8447-42a0-9261-f96c52c71e6d" containerName="revoke-keys"
Oct 30 07:49:55.497080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.497023    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8281e1-8447-42a0-9261-f96c52c71e6d" containerName="revoke-keys"
Oct 30 07:49:55.497196 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.497186    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8281e1-8447-42a0-9261-f96c52c71e6d" containerName="revoke-keys"
Oct 30 07:49:55.500310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.500278    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.519413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.519372    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"]
Oct 30 07:49:55.524795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.524767    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:49:55.524977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.524873    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:49:55.640500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.640461    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.640719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.640521    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.640719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.640647    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvpk\" (UniqueName: \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.742309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.742205    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.742527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.742398    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.742527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.742511    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvpk\" (UniqueName: \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.743087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.743057    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.744654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.744634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.766262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.766193    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvpk\" (UniqueName: \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\") pod \"maas-api-revoke-sub-19309ad2-bnggm\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:55.816484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:55.816450    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:56.127217 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:49:56.127191    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6777eab_f983_48d8_941f_29877ad83b42.slice/crio-6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a WatchSource:0}: Error finding container 6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a: Status 404 returned error can't find the container with id 6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a
Oct 30 07:49:56.127849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:56.127826    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"]
Oct 30 07:49:56.174161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:56.174125    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm" event={"ID":"c6777eab-f983-48d8-941f-29877ad83b42","Type":"ContainerStarted","Data":"6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a"}
Oct 30 07:49:56.320270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:56.320236    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:57.185550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:57.185515    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm" event={"ID":"c6777eab-f983-48d8-941f-29877ad83b42","Type":"ContainerStarted","Data":"02a11d057db9fd55a76eed7ad3e2d01c07610f8075243606869b0bc0155e8958"}
Oct 30 07:49:57.227496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:57.227439    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm" podStartSLOduration=2.227423254 podStartE2EDuration="2.227423254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:57.225572843 +0000 UTC m=+3157.658336736" watchObservedRunningTime="2026-10-30 07:49:57.227423254 +0000 UTC m=+3157.660187146"
Oct 30 07:49:58.086427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:58.086393    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:49:58.203962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:58.203931    6738 generic.go:356] "Generic (PLEG): container finished" podID="c6777eab-f983-48d8-941f-29877ad83b42" containerID="02a11d057db9fd55a76eed7ad3e2d01c07610f8075243606869b0bc0155e8958" exitCode=0
Oct 30 07:49:58.204438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:58.204005    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm" event={"ID":"c6777eab-f983-48d8-941f-29877ad83b42","Type":"ContainerDied","Data":"02a11d057db9fd55a76eed7ad3e2d01c07610f8075243606869b0bc0155e8958"}
Oct 30 07:49:59.385163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.385134    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:49:59.504670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.504582    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\") pod \"c6777eab-f983-48d8-941f-29877ad83b42\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") "
Oct 30 07:49:59.504670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.504643    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\") pod \"c6777eab-f983-48d8-941f-29877ad83b42\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") "
Oct 30 07:49:59.504897 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.504820    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\" (UniqueName: \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\") pod \"c6777eab-f983-48d8-941f-29877ad83b42\" (UID: \"c6777eab-f983-48d8-941f-29877ad83b42\") "
Oct 30 07:49:59.505008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.504985    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca" pod "c6777eab-f983-48d8-941f-29877ad83b42" (UID: "c6777eab-f983-48d8-941f-29877ad83b42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:49:59.505473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.505452    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:49:59.506824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.506801    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca" pod "c6777eab-f983-48d8-941f-29877ad83b42" (UID: "c6777eab-f983-48d8-941f-29877ad83b42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:49:59.507070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.507052    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk" pod "c6777eab-f983-48d8-941f-29877ad83b42" (UID: "c6777eab-f983-48d8-941f-29877ad83b42"). InnerVolumeSpecName "kube-api-access-6qvpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:49:59.606722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.606687    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvpk\" (UniqueName: \"kubernetes.io/projected/c6777eab-f983-48d8-941f-29877ad83b42-kube-api-access-6qvpk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:49:59.606722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:49:59.606719    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6777eab-f983-48d8-941f-29877ad83b42-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:00.054419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:00.054380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:00.223384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:00.223352    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"
Oct 30 07:50:00.223606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:00.223350    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm" event={"ID":"c6777eab-f983-48d8-941f-29877ad83b42","Type":"ContainerDied","Data":"6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a"}
Oct 30 07:50:00.223606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:00.223494    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8754ec7830c4a546d220c89d805de12a418dd26712ec207eb18e072ea3672a"
Oct 30 07:50:01.629255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:01.629211    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:50:01.629711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:01.629477    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy" containerID="cri-o://bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57" gracePeriod=30
Oct 30 07:50:01.855486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:01.855434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:03.902283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:03.902245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:05.714574 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:05.714516    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:06.936020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:06.935988    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:50:07.066141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066098    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066180    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066347    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066427    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066460    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066493    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066508    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:07.066731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066585    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066699    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:07.066852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066727    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.066852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066727    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:07.066852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066813    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:07.066852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.066821    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\") pod \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\" (UID: \"9e71af10-fc20-4c4e-8458-93b5f3afd68e\") "
Oct 30 07:50:07.067155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067135    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:07.067634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067613    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.067728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067638    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.067728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067650    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.067728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067663    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.067728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.067674    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.068946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.068921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:50:07.069061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.069044    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:07.069130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.069071    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "kube-api-access-dzj6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:07.069185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.069122    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy" pod "9e71af10-fc20-4c4e-8458-93b5f3afd68e" (UID: "9e71af10-fc20-4c4e-8458-93b5f3afd68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:07.169884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.169792    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzj6g\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-kube-api-access-dzj6g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.169884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.169829    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.169884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.169841    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.169884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.169853    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e71af10-fc20-4c4e-8458-93b5f3afd68e-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:07.292918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.288433    6738 generic.go:356] "Generic (PLEG): container finished" podID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerID="bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57" exitCode=0
Oct 30 07:50:07.292918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.288522    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" event={"ID":"9e71af10-fc20-4c4e-8458-93b5f3afd68e","Type":"ContainerDied","Data":"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"}
Oct 30 07:50:07.292918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.288592    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l" event={"ID":"9e71af10-fc20-4c4e-8458-93b5f3afd68e","Type":"ContainerDied","Data":"5b89ea7049696937c532f9afd3eb519ba77e463f53cb59fd4620df5ca10d3766"}
Oct 30 07:50:07.292918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.288639    6738 scope.go:122] "RemoveContainer" containerID="bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"
Oct 30 07:50:07.292918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.288866    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"
Oct 30 07:50:07.306277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.306258    6738 scope.go:122] "RemoveContainer" containerID="bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"
Oct 30 07:50:07.306711 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:50:07.306684    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57\": container with ID starting with bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57 not found: ID does not exist" containerID="bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"
Oct 30 07:50:07.306781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.306724    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57"} err="failed to get container status \"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57\": rpc error: code = NotFound desc = could not find container \"bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57\": container with ID starting with bdfaae913dacbfe6e28a79cade8b83cdc69320869920912d03bb306522307d57 not found: ID does not exist"
Oct 30 07:50:07.346602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.346569    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:50:07.351381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.351348    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-4jq7l"]
Oct 30 07:50:07.493414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.493288    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:07.522177 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:50:07.522141    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:50:07.693068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:07.693033    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" path="/var/lib/kubelet/pods/9e71af10-fc20-4c4e-8458-93b5f3afd68e/volumes"
Oct 30 07:50:09.267540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:09.267496    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:11.041097 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:11.041056    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:12.272419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.272381    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:12.614374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.614248    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"]
Oct 30 07:50:12.615197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615173    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c6777eab-f983-48d8-941f-29877ad83b42" containerName="revoke-keys"
Oct 30 07:50:12.615197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615195    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6777eab-f983-48d8-941f-29877ad83b42" containerName="revoke-keys"
Oct 30 07:50:12.615430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615236    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy"
Oct 30 07:50:12.615430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615245    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy"
Oct 30 07:50:12.615509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615455    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6777eab-f983-48d8-941f-29877ad83b42" containerName="revoke-keys"
Oct 30 07:50:12.615509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.615472    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e71af10-fc20-4c4e-8458-93b5f3afd68e" containerName="istio-proxy"
Oct 30 07:50:12.619424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.619401    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.636433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.636398    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"]
Oct 30 07:50:12.641691 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.641658    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:50:12.641872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.641770    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:50:12.786728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.786682    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xxk\" (UniqueName: \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.787055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.787029    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.787147 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.787133    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.888809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.888777    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.889008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.888843    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.889008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.888960    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xxk\" (UniqueName: \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.889656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.889629    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.891245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.891221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.912826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.912790    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xxk\" (UniqueName: \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\") pod \"maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:12.936658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:12.936619    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:13.072031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:13.071994    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:13.246881 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:50:13.246843    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522bd838_32e7_42c7_bc17_ded99739a112.slice/crio-dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b WatchSource:0}: Error finding container dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b: Status 404 returned error can't find the container with id dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b
Oct 30 07:50:13.249110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:13.249084    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"]
Oct 30 07:50:13.433227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:13.433191    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4" event={"ID":"522bd838-32e7-42c7-bc17-ded99739a112","Type":"ContainerStarted","Data":"dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b"}
Oct 30 07:50:14.445382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:14.445332    6738 generic.go:356] "Generic (PLEG): container finished" podID="522bd838-32e7-42c7-bc17-ded99739a112" containerID="69bdd75c0a9ba1a4404a847cc7d5b295e040d4438eb1cb853ecbee026df1b926" exitCode=0
Oct 30 07:50:14.445802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:14.445413    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4" event={"ID":"522bd838-32e7-42c7-bc17-ded99739a112","Type":"ContainerDied","Data":"69bdd75c0a9ba1a4404a847cc7d5b295e040d4438eb1cb853ecbee026df1b926"}
Oct 30 07:50:15.065703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.065636    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:15.220082 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.220038    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:15.642334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.642289    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:15.744148 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.744057    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\") pod \"522bd838-32e7-42c7-bc17-ded99739a112\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") "
Oct 30 07:50:15.744334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.744210    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\" (UniqueName: \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\") pod \"522bd838-32e7-42c7-bc17-ded99739a112\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") "
Oct 30 07:50:15.744441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.744426    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\") pod \"522bd838-32e7-42c7-bc17-ded99739a112\" (UID: \"522bd838-32e7-42c7-bc17-ded99739a112\") "
Oct 30 07:50:15.744836 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.744806    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca" pod "522bd838-32e7-42c7-bc17-ded99739a112" (UID: "522bd838-32e7-42c7-bc17-ded99739a112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:15.745375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.745352    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:15.746709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.746684    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca" pod "522bd838-32e7-42c7-bc17-ded99739a112" (UID: "522bd838-32e7-42c7-bc17-ded99739a112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:50:15.746805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.746731    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk" pod "522bd838-32e7-42c7-bc17-ded99739a112" (UID: "522bd838-32e7-42c7-bc17-ded99739a112"). InnerVolumeSpecName "kube-api-access-p6xxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:15.847202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.847164    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xxk\" (UniqueName: \"kubernetes.io/projected/522bd838-32e7-42c7-bc17-ded99739a112-kube-api-access-p6xxk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:15.847202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:15.847199    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522bd838-32e7-42c7-bc17-ded99739a112-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:16.465602 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:16.465567    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"
Oct 30 07:50:16.465799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:16.465597    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4" event={"ID":"522bd838-32e7-42c7-bc17-ded99739a112","Type":"ContainerDied","Data":"dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b"}
Oct 30 07:50:16.465799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:16.465645    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea3aae8f1b2022c7598b5cb72cc458ac532fc7a71b7413a152364d9d969367b"
Oct 30 07:50:17.050458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:17.050379    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:18.949347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:18.949301    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:20.854527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:20.854489    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:22.811838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:22.811802    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:23.158530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:23.158474    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:50:23.353141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:23.337610    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:23.577381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:23.559591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:24.127034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:24.126985    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:50:24.135104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:24.135065    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" containerID="cri-o://280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434" gracePeriod=30
Oct 30 07:50:25.031214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:25.031174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:27.043948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:27.043912    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:28.032208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:28.032173    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"]
Oct 30 07:50:28.043832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:28.043793    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e073a368-zvzzm"]
Oct 30 07:50:28.160391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:28.160344    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing" containerID="cri-o://8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28" gracePeriod=25
Oct 30 07:50:28.933193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:28.933156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:29.011044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.010994    6738 patch_prober.go:29] interesting pod/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:50:29.011404 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.011367    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:50:29.451069 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.451041    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:50:29.598375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598228    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598328    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598447    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598643 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598590    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598642    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598699    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598679    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:29.598940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598779    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.598940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598853    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:29.598940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598875    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:29.598940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598902    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.599156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.598947    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\") pod \"204e290c-60b9-4347-a3ef-ca6242c0cf75\" (UID: \"204e290c-60b9-4347-a3ef-ca6242c0cf75\") "
Oct 30 07:50:29.599214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599146    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:29.599681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599652    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e290c-60b9-4347-a3ef-ca6242c0cf75-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.599831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599708    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.599831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599725    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.599831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599727    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:29.599831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.599740    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.601010 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.600980    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:29.601377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.601346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:50:29.601494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.601439    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:29.601494 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.601452    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q" pod "204e290c-60b9-4347-a3ef-ca6242c0cf75" (UID: "204e290c-60b9-4347-a3ef-ca6242c0cf75"). InnerVolumeSpecName "kube-api-access-pf97q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:29.694125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.694083    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c243912-ba27-4d8f-919b-ef256772a49b" path="/var/lib/kubelet/pods/6c243912-ba27-4d8f-919b-ef256772a49b/volumes"
Oct 30 07:50:29.701139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.701108    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.701139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.701135    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf97q\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-kube-api-access-pf97q\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.701396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.701148    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.701396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.701156    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.701396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.701164    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e290c-60b9-4347-a3ef-ca6242c0cf75-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:29.824693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.824656    6738 generic.go:356] "Generic (PLEG): container finished" podID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerID="280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434" exitCode=0
Oct 30 07:50:29.824872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.824713    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"
Oct 30 07:50:29.824872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.824740    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" event={"ID":"204e290c-60b9-4347-a3ef-ca6242c0cf75","Type":"ContainerDied","Data":"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"}
Oct 30 07:50:29.824872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.824810    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7" event={"ID":"204e290c-60b9-4347-a3ef-ca6242c0cf75","Type":"ContainerDied","Data":"ec0242f2dbd524a67c79ad1b45adc06ae536d93db244dbc8a03da2e664f0c260"}
Oct 30 07:50:29.824872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.824841    6738 scope.go:122] "RemoveContainer" containerID="280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"
Oct 30 07:50:29.837707 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.837685    6738 scope.go:122] "RemoveContainer" containerID="280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"
Oct 30 07:50:29.838125 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:50:29.838105    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434\": container with ID starting with 280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434 not found: ID does not exist" containerID="280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"
Oct 30 07:50:29.838224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.838137    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434"} err="failed to get container status \"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434\": rpc error: code = NotFound desc = could not find container \"280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434\": container with ID starting with 280dc49d2ca4b44581da59fece087c890da0e45e2faf6d7c88779f157951c434 not found: ID does not exist"
Oct 30 07:50:29.881245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.881207    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:50:29.893918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:29.893883    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-06e19a-openshift-default-6ddddc4467-7psh7"]
Oct 30 07:50:30.385621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:30.385590    6738 scope.go:122] "RemoveContainer" containerID="4c23f1fd9ab5824c73eafbb8ad3f55a1210dbc43dc54cc6bc3902ffa6148b6f9"
Oct 30 07:50:30.397222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:30.397198    6738 scope.go:122] "RemoveContainer" containerID="d8c64d6cac4e10eaef723783ae59195116c04875f4c9c8fd8f3ee7c076de5940"
Oct 30 07:50:30.871811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:30.871670    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:31.693084 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:31.693040    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" path="/var/lib/kubelet/pods/204e290c-60b9-4347-a3ef-ca6242c0cf75/volumes"
Oct 30 07:50:32.118405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:32.118287    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:50:32.652565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:32.652528    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:34.181405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:34.181365    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:34.700128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:34.700091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:37.522093 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:50:37.522055    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:50:37.597873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:37.597833    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:37.834879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:37.834785    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:38.728749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:38.728708    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:39.032766 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:39.032672    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"]
Oct 30 07:50:39.044575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:39.044540    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d174c9a-vwgjl"]
Oct 30 07:50:39.692518 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:39.692483    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab" path="/var/lib/kubelet/pods/f8ddf8c4-0fcb-41ae-9a52-eef5bbb7bbab/volumes"
Oct 30 07:50:40.161004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:40.160965    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:40.657123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:40.657092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:42.117450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:42.117409    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:50:42.727114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:42.727076    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:44.483633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:44.483591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:46.221180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:46.221130    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:48.059416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.059377    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:48.169621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.169527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:48.490237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.490207    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:50:48.566337 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.566261    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\") pod \"94d81899-c2b9-4536-9a5b-0202cc6dff08\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") "
Oct 30 07:50:48.566559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.566414    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\" (UniqueName: \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\") pod \"94d81899-c2b9-4536-9a5b-0202cc6dff08\" (UID: \"94d81899-c2b9-4536-9a5b-0202cc6dff08\") "
Oct 30 07:50:48.566645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.566618    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume" pod "94d81899-c2b9-4536-9a5b-0202cc6dff08" (UID: "94d81899-c2b9-4536-9a5b-0202cc6dff08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:48.566976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.566957    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d81899-c2b9-4536-9a5b-0202cc6dff08-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:48.568705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.568677    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k" pod "94d81899-c2b9-4536-9a5b-0202cc6dff08" (UID: "94d81899-c2b9-4536-9a5b-0202cc6dff08"). InnerVolumeSpecName "kube-api-access-cc84k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:48.669566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:48.669521    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc84k\" (UniqueName: \"kubernetes.io/projected/94d81899-c2b9-4536-9a5b-0202cc6dff08-kube-api-access-cc84k\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:50:49.040994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.040900    6738 generic.go:356] "Generic (PLEG): container finished" podID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerID="8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28" exitCode=0
Oct 30 07:50:49.040994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.040957    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"
Oct 30 07:50:49.040994 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.040983    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" event={"ID":"94d81899-c2b9-4536-9a5b-0202cc6dff08","Type":"ContainerDied","Data":"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"}
Oct 30 07:50:49.041285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.041043    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24" event={"ID":"94d81899-c2b9-4536-9a5b-0202cc6dff08","Type":"ContainerDied","Data":"008375da9d9ae377791b904e4e9bbf973b2dee9dd21815a36d59c625d1264b88"}
Oct 30 07:50:49.041285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.041069    6738 scope.go:122] "RemoveContainer" containerID="8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"
Oct 30 07:50:49.053109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.053085    6738 scope.go:122] "RemoveContainer" containerID="8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"
Oct 30 07:50:49.053512 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:50:49.053469    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28\": container with ID starting with 8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28 not found: ID does not exist" containerID="8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"
Oct 30 07:50:49.053624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.053517    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28"} err="failed to get container status \"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28\": rpc error: code = NotFound desc = could not find container \"8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28\": container with ID starting with 8c56dfec217bc963f519e49f0f100927888691f249be2dcd4aeca7771b160f28 not found: ID does not exist"
Oct 30 07:50:49.096428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.096390    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:50:49.110456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.110417    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad434e-566f84994c-wnk24"]
Oct 30 07:50:49.692942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.692909    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" path="/var/lib/kubelet/pods/94d81899-c2b9-4536-9a5b-0202cc6dff08/volumes"
Oct 30 07:50:49.886360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:49.886321    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:51.728664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:51.728628    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:53.685177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:53.685138    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:55.033352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:55.033285    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"]
Oct 30 07:50:55.091878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:55.085802    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40183da1-z49mm"]
Oct 30 07:50:55.693289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:55.693256    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2042562-74a6-496c-8524-3b80d8892414" path="/var/lib/kubelet/pods/f2042562-74a6-496c-8524-3b80d8892414/volumes"
Oct 30 07:50:55.730194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:55.730162    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:57.345619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:57.345552    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:58.076503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.076464    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:58.191620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.191586    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:58.273427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.273395    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"]
Oct 30 07:50:58.288181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.288143    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ad434e-xnmf4"]
Oct 30 07:50:58.324938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.324770    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:58.796089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:58.796050    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:50:59.694079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:50:59.694044    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522bd838-32e7-42c7-bc17-ded99739a112" path="/var/lib/kubelet/pods/522bd838-32e7-42c7-bc17-ded99739a112/volumes"
Oct 30 07:51:00.227729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:00.227693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:02.049394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:02.049356    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:03.937168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:03.937129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:05.253821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:05.253784    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:05.364941 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:05.364898    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:05.612128 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:05.612037    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:05.805272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:05.805103    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:06.330754 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.330713    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"]
Oct 30 07:51:06.331619 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331602    6738 cpu_manager.go:414] "containerMap: removing container" podUID="522bd838-32e7-42c7-bc17-ded99739a112" containerName="revoke-keys"
Oct 30 07:51:06.331678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331622    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522bd838-32e7-42c7-bc17-ded99739a112" containerName="revoke-keys"
Oct 30 07:51:06.331678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331664    6738 cpu_manager.go:414] "containerMap: removing container" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing"
Oct 30 07:51:06.331678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331671    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing"
Oct 30 07:51:06.331784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331699    6738 cpu_manager.go:414] "containerMap: removing container" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy"
Oct 30 07:51:06.331784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331704    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy"
Oct 30 07:51:06.331854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331818    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="204e290c-60b9-4347-a3ef-ca6242c0cf75" containerName="istio-proxy"
Oct 30 07:51:06.331854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331831    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="522bd838-32e7-42c7-bc17-ded99739a112" containerName="revoke-keys"
Oct 30 07:51:06.331854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.331844    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d81899-c2b9-4536-9a5b-0202cc6dff08" containerName="payload-pre-processing"
Oct 30 07:51:06.337346 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.337283    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.354124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.354097    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"]
Oct 30 07:51:06.375596 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.375559    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.375918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.375813    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgbl\" (UniqueName: \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.376026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.375939    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.381536 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.381506    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:51:06.381737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.381676    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:51:06.477992 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.477954    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.478177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.478066    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.478254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.478234    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgbl\" (UniqueName: \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.478872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.478850    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.480402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.480381    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.502120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.502089    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgbl\" (UniqueName: \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\") pod \"maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.654181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.654143    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:06.974962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:06.974929    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"]
Oct 30 07:51:06.975458 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:51:06.975434    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee92d8b_feaa_4ee4_b98a_4f4a4719dcf3.slice/crio-80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587 WatchSource:0}: Error finding container 80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587: Status 404 returned error can't find the container with id 80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587
Oct 30 07:51:07.031586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:07.031556    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"]
Oct 30 07:51:07.035811 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:07.035783    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa90ed1f-dpl6j"]
Oct 30 07:51:07.229562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:07.229465    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49" event={"ID":"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3","Type":"ContainerStarted","Data":"80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587"}
Oct 30 07:51:07.522781 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:51:07.522743    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:51:07.693624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:07.693591    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c64aa94-7e8f-4e26-919d-234e9df628a1" path="/var/lib/kubelet/pods/8c64aa94-7e8f-4e26-919d-234e9df628a1/volumes"
Oct 30 07:51:07.989459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:07.989407    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:08.107562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:08.107522    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:08.240577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:08.240477    6738 generic.go:356] "Generic (PLEG): container finished" podID="9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" containerID="048e2ecc34e6ff7b63a8befb6c05e586810b0324aac85cb49afb12a6207e8876" exitCode=0
Oct 30 07:51:08.240577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:08.240560    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49" event={"ID":"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3","Type":"ContainerDied","Data":"048e2ecc34e6ff7b63a8befb6c05e586810b0324aac85cb49afb12a6207e8876"}
Oct 30 07:51:09.450709 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.450674    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:09.546261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.546139    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\" (UniqueName: \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\") pod \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") "
Oct 30 07:51:09.546261 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.546187    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\") pod \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") "
Oct 30 07:51:09.546620 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.546601    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\") pod \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\" (UID: \"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3\") "
Oct 30 07:51:09.547688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.547659    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca" pod "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" (UID: "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:51:09.550007 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.549978    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca" pod "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" (UID: "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:51:09.550140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.550115    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl" pod "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" (UID: "9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3"). InnerVolumeSpecName "kube-api-access-ghgbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:09.648823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.648767    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:09.648823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.648815    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgbl\" (UniqueName: \"kubernetes.io/projected/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-kube-api-access-ghgbl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:09.648823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.648826    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:09.857111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:09.857018    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:10.261456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:10.261418    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49" event={"ID":"9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3","Type":"ContainerDied","Data":"80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587"}
Oct 30 07:51:10.261654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:10.261481    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80a9f292970537b2741ecfecb25c7c2ad73556b8928f5fe2c4be31e638286587"
Oct 30 07:51:10.261654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:10.261430    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"
Oct 30 07:51:11.680499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:11.680461    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:13.524114 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:13.524075    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:15.504316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:15.495718    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:16.864528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:16.864490    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:51:16.966272 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:16.966234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:17.108314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:17.104267    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:17.305965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:17.305925    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:19.033918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.033883    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"]
Oct 30 07:51:19.038175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.038138    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a198e6-7tdqg"]
Oct 30 07:51:19.085760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.085718    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:19.473307 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.473270    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"]
Oct 30 07:51:19.474552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.474527    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" containerName="revoke-keys"
Oct 30 07:51:19.474552 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.474551    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" containerName="revoke-keys"
Oct 30 07:51:19.474791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.474774    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" containerName="revoke-keys"
Oct 30 07:51:19.479471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.479450    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.503335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.503261    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"]
Oct 30 07:51:19.509358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.509265    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:51:19.509557 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.509414    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:51:19.543621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.543582    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.543931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.543903    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.544003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.543985    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxp6\" (UniqueName: \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.646445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.646383    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.646668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.646648    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.646743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.646719    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxp6\" (UniqueName: \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.647415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.647390    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.648781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.648748    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.670236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.670205    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxp6\" (UniqueName: \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\") pod \"maas-api-revoke-sub-fe4f433b-rf6bk\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:19.700464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.700422    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534f9eef-4c59-456b-bd56-9eeac8990c24" path="/var/lib/kubelet/pods/534f9eef-4c59-456b-bd56-9eeac8990c24/volumes"
Oct 30 07:51:19.809055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.808975    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:51:19.815239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:19.815214    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:20.132831 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:51:20.132787    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda28ff55_b341_4c3e_8cbc_4afce1dccf01.slice/crio-6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b WatchSource:0}: Error finding container 6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b: Status 404 returned error can't find the container with id 6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b
Oct 30 07:51:20.133359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:20.133329    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"]
Oct 30 07:51:20.352125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:20.352082    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk" event={"ID":"da28ff55-b341-4c3e-8cbc-4afce1dccf01","Type":"ContainerStarted","Data":"6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b"}
Oct 30 07:51:20.999472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:20.999434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:21.362256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:21.362161    6738 generic.go:356] "Generic (PLEG): container finished" podID="da28ff55-b341-4c3e-8cbc-4afce1dccf01" containerID="42d2322054fc66880bfe637774fb597cd09b938cba1ec1b9238436887f1876b9" exitCode=0
Oct 30 07:51:21.362636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:21.362250    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk" event={"ID":"da28ff55-b341-4c3e-8cbc-4afce1dccf01","Type":"ContainerDied","Data":"42d2322054fc66880bfe637774fb597cd09b938cba1ec1b9238436887f1876b9"}
Oct 30 07:51:21.865394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:21.865351    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing" containerID="cri-o://5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50" gracePeriod=25
Oct 30 07:51:22.545026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.544996    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:22.698670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.698580    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\") pod \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") "
Oct 30 07:51:22.698841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.698726    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\" (UniqueName: \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\") pod \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") "
Oct 30 07:51:22.698841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.698761    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\") pod \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\" (UID: \"da28ff55-b341-4c3e-8cbc-4afce1dccf01\") "
Oct 30 07:51:22.699174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.699151    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca" pod "da28ff55-b341-4c3e-8cbc-4afce1dccf01" (UID: "da28ff55-b341-4c3e-8cbc-4afce1dccf01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:51:22.699519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.699500    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:22.700806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.700778    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca" pod "da28ff55-b341-4c3e-8cbc-4afce1dccf01" (UID: "da28ff55-b341-4c3e-8cbc-4afce1dccf01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:51:22.700920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.700905    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6" pod "da28ff55-b341-4c3e-8cbc-4afce1dccf01" (UID: "da28ff55-b341-4c3e-8cbc-4afce1dccf01"). InnerVolumeSpecName "kube-api-access-ghxp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:22.800814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.800767    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghxp6\" (UniqueName: \"kubernetes.io/projected/da28ff55-b341-4c3e-8cbc-4afce1dccf01-kube-api-access-ghxp6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:22.800814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.800802    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da28ff55-b341-4c3e-8cbc-4afce1dccf01-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:22.917705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:22.917670    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:23.385259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:23.385229    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"
Oct 30 07:51:23.385468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:23.385253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk" event={"ID":"da28ff55-b341-4c3e-8cbc-4afce1dccf01","Type":"ContainerDied","Data":"6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b"}
Oct 30 07:51:23.385468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:23.385339    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bfaf3eaca0043b7e1713953665c905727a64e7644b9cbdb262b82907355d24b"
Oct 30 07:51:23.804660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:23.804555    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:51:24.795095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:24.795044    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:26.870932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:26.870895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:28.657014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:28.656971    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:28.921248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:28.921151    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:30.034326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:30.034272    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"]
Oct 30 07:51:30.046752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:30.046712    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8e84d3-9t7bw"]
Oct 30 07:51:30.453321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:30.453274    6738 scope.go:122] "RemoveContainer" containerID="4d611f5a3c46f0c6448c6cfec0c0df3414d8cf2e405413d2d8a16e0ef7581462"
Oct 30 07:51:30.604640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:30.604591    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:31.692693 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:31.692656    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5de333b-2bfc-4b24-9e7a-dd55aa2da323" path="/var/lib/kubelet/pods/d5de333b-2bfc-4b24-9e7a-dd55aa2da323/volumes"
Oct 30 07:51:32.384573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:32.384533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:33.329801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:33.329761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:33.805581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:33.805543    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:51:34.242146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:34.242097    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:34.353627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:34.353587    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:36.032364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:36.032284    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:37.526514 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:51:37.526468    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:51:37.862603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:37.862512    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:39.637879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:39.637837    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:42.071153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.071080    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:42.218615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.218587    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:51:42.396636 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.396601    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\") pod \"3882ab1d-f3f6-4344-95bf-45474a27e29a\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") "
Oct 30 07:51:42.396835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.396737    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\") pod \"3882ab1d-f3f6-4344-95bf-45474a27e29a\" (UID: \"3882ab1d-f3f6-4344-95bf-45474a27e29a\") "
Oct 30 07:51:42.397123 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.397100    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume" pod "3882ab1d-f3f6-4344-95bf-45474a27e29a" (UID: "3882ab1d-f3f6-4344-95bf-45474a27e29a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:51:42.397484 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.397465    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3882ab1d-f3f6-4344-95bf-45474a27e29a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:42.398887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.398861    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx" pod "3882ab1d-f3f6-4344-95bf-45474a27e29a" (UID: "3882ab1d-f3f6-4344-95bf-45474a27e29a"). InnerVolumeSpecName "kube-api-access-s64bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:42.499703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.499609    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64bx\" (UniqueName: \"kubernetes.io/projected/3882ab1d-f3f6-4344-95bf-45474a27e29a-kube-api-access-s64bx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:42.589741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.589701    6738 generic.go:356] "Generic (PLEG): container finished" podID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerID="5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50" exitCode=0
Oct 30 07:51:42.589935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.589747    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"
Oct 30 07:51:42.589935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.589781    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" event={"ID":"3882ab1d-f3f6-4344-95bf-45474a27e29a","Type":"ContainerDied","Data":"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"}
Oct 30 07:51:42.589935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.589834    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv" event={"ID":"3882ab1d-f3f6-4344-95bf-45474a27e29a","Type":"ContainerDied","Data":"f53a97fb9e49d608a228f2f2376143afbbe10eaf2ad25b6014958cb8f72233ed"}
Oct 30 07:51:42.589935 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.589871    6738 scope.go:122] "RemoveContainer" containerID="5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"
Oct 30 07:51:42.602626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.602598    6738 scope.go:122] "RemoveContainer" containerID="5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"
Oct 30 07:51:42.602927 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:51:42.602905    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50\": container with ID starting with 5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50 not found: ID does not exist" containerID="5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"
Oct 30 07:51:42.603020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.602940    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50"} err="failed to get container status \"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50\": rpc error: code = NotFound desc = could not find container \"5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50\": container with ID starting with 5f4b23ddb71231a464c893179a17b2d594a30cb175b31a9d000eaae88795cc50 not found: ID does not exist"
Oct 30 07:51:42.645604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.645567    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:51:42.650791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:42.650757    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4ab37e-cdb874bf5-frpjv"]
Oct 30 07:51:43.392458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:43.392424    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:43.693270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:43.693173    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" path="/var/lib/kubelet/pods/3882ab1d-f3f6-4344-95bf-45474a27e29a/volumes"
Oct 30 07:51:44.071250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.071174    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"]
Oct 30 07:51:44.072050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072033    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da28ff55-b341-4c3e-8cbc-4afce1dccf01" containerName="revoke-keys"
Oct 30 07:51:44.072098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072052    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da28ff55-b341-4c3e-8cbc-4afce1dccf01" containerName="revoke-keys"
Oct 30 07:51:44.072098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072079    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing"
Oct 30 07:51:44.072098 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072085    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing"
Oct 30 07:51:44.072229 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072220    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da28ff55-b341-4c3e-8cbc-4afce1dccf01" containerName="revoke-keys"
Oct 30 07:51:44.072270 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.072234    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3882ab1d-f3f6-4344-95bf-45474a27e29a" containerName="payload-pre-processing"
Oct 30 07:51:44.077710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.077686    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.093095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.093067    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"]
Oct 30 07:51:44.100438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.100405    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:51:44.100628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.100508    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:51:44.135809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.135758    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.136042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.135835    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pw9\" (UniqueName: \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.136209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.136186    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.164539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.162937    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:44.237648 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.237609    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.237847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.237763    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.237847 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.237806    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pw9\" (UniqueName: \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.238587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.238560    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.240343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.240323    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.262222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.262187    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pw9\" (UniqueName: \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\") pod \"maas-api-revoke-sub-b0617966-vpc7k\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.395727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.395693    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:44.706458 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:51:44.706429    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde995859_a91b_41e3_8eb5_32fb607639d5.slice/crio-a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72 WatchSource:0}: Error finding container a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72: Status 404 returned error can't find the container with id a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72
Oct 30 07:51:44.707745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:44.707716    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"]
Oct 30 07:51:45.618481 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:45.618442    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k" event={"ID":"de995859-a91b-41e3-8eb5-32fb607639d5","Type":"ContainerStarted","Data":"eec3d0ff11821d6bce19443886beb5d1c2896ecc6bf472fa0dc8d00f94e60b0c"}
Oct 30 07:51:45.618972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:45.618498    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k" event={"ID":"de995859-a91b-41e3-8eb5-32fb607639d5","Type":"ContainerStarted","Data":"a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72"}
Oct 30 07:51:45.662169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:45.662096    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k" podStartSLOduration=1.6620742659999999 podStartE2EDuration="1.662074266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:45.659124195 +0000 UTC m=+3266.091888089" watchObservedRunningTime="2026-10-30 07:51:45.662074266 +0000 UTC m=+3266.094838159"
Oct 30 07:51:45.955986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:45.955948    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:46.629189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:46.629151    6738 generic.go:356] "Generic (PLEG): container finished" podID="de995859-a91b-41e3-8eb5-32fb607639d5" containerID="eec3d0ff11821d6bce19443886beb5d1c2896ecc6bf472fa0dc8d00f94e60b0c" exitCode=0
Oct 30 07:51:46.629625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:46.629234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k" event={"ID":"de995859-a91b-41e3-8eb5-32fb607639d5","Type":"ContainerDied","Data":"eec3d0ff11821d6bce19443886beb5d1c2896ecc6bf472fa0dc8d00f94e60b0c"}
Oct 30 07:51:47.834253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.834227    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:47.849509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.849478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:47.921377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.913890    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\") pod \"de995859-a91b-41e3-8eb5-32fb607639d5\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") "
Oct 30 07:51:47.921377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.914022    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\") pod \"de995859-a91b-41e3-8eb5-32fb607639d5\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") "
Oct 30 07:51:47.921377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.914117    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\" (UniqueName: \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\") pod \"de995859-a91b-41e3-8eb5-32fb607639d5\" (UID: \"de995859-a91b-41e3-8eb5-32fb607639d5\") "
Oct 30 07:51:47.935975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.935499    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca" pod "de995859-a91b-41e3-8eb5-32fb607639d5" (UID: "de995859-a91b-41e3-8eb5-32fb607639d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:51:47.937169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.937090    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:47.938496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.938463    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca" pod "de995859-a91b-41e3-8eb5-32fb607639d5" (UID: "de995859-a91b-41e3-8eb5-32fb607639d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:51:47.952572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:47.952530    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9" pod "de995859-a91b-41e3-8eb5-32fb607639d5" (UID: "de995859-a91b-41e3-8eb5-32fb607639d5"). InnerVolumeSpecName "kube-api-access-z9pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:48.039548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:48.039510    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de995859-a91b-41e3-8eb5-32fb607639d5-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:48.039548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:48.039544    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9pw9\" (UniqueName: \"kubernetes.io/projected/de995859-a91b-41e3-8eb5-32fb607639d5-kube-api-access-z9pw9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:51:48.650226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:48.650190    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"
Oct 30 07:51:48.650226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:48.650211    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k" event={"ID":"de995859-a91b-41e3-8eb5-32fb607639d5","Type":"ContainerDied","Data":"a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72"}
Oct 30 07:51:48.650500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:48.650258    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0440f539ba1a07ab481c85a908cf07db483a0d3f2a2c83c8c90560027d97c72"
Oct 30 07:51:48.719214 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:51:48.719182    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde995859_a91b_41e3_8eb5_32fb607639d5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 07:51:49.680466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:49.680309    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:51.563804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:51.563767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:53.352029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:53.351986    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:55.238311 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:55.238256    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:57.028951 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:57.028918    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:51:58.959772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:51:58.959739    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:00.846986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:00.846943    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:02.284418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:02.284375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:02.700468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:02.700427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:04.586958 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:04.586920    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:06.469340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:06.469248    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:07.521569 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:07.521530    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:52:08.487456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:08.487418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:10.084635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:10.084597    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:10.875375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:10.875337    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:10.998635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:10.998584    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:11.055286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.055248    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"]
Oct 30 07:52:11.067152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.067115    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ab37e-cxv49"]
Oct 30 07:52:11.116186 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.116109    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:11.409405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.409372    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:11.518036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.517984    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:11.693577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:11.693496    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3" path="/var/lib/kubelet/pods/9ee92d8b-feaa-4ee4-b98a-4f4a4719dcf3/volumes"
Oct 30 07:52:12.290535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.290495    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:12.511942 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.511908    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"]
Oct 30 07:52:12.512855 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.512836    6738 cpu_manager.go:414] "containerMap: removing container" podUID="de995859-a91b-41e3-8eb5-32fb607639d5" containerName="revoke-keys"
Oct 30 07:52:12.512919 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.512856    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de995859-a91b-41e3-8eb5-32fb607639d5" containerName="revoke-keys"
Oct 30 07:52:12.512995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.512986    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="de995859-a91b-41e3-8eb5-32fb607639d5" containerName="revoke-keys"
Oct 30 07:52:12.517527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.517504    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.533650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.533622    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"]
Oct 30 07:52:12.540515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.540482    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:52:12.540822 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.540538    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:52:12.632734 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.632700    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.632972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.632871    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.633241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.633218    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9kq\" (UniqueName: \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.735354 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.735272    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.735553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.735420    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.735553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.735498    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9kq\" (UniqueName: \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.736119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.736093    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.737908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.737883    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.759215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.759177    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9kq\" (UniqueName: \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:12.834074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:12.833985    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:13.150230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:13.150195    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"]
Oct 30 07:52:13.150504 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:52:13.150479    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbed573_9d8d_4bfd_919d_4036986cc3b1.slice/crio-d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0 WatchSource:0}: Error finding container d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0: Status 404 returned error can't find the container with id d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0
Oct 30 07:52:13.280443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:13.280401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:13.920818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:13.920779    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv" event={"ID":"2bbed573-9d8d-4bfd-919d-4036986cc3b1","Type":"ContainerStarted","Data":"af1d2399eeefd61642b6cacfa4f06ea289d908a5f2ca6e36af4c695bd65a411a"}
Oct 30 07:52:13.921267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:13.920841    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv" event={"ID":"2bbed573-9d8d-4bfd-919d-4036986cc3b1","Type":"ContainerStarted","Data":"d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0"}
Oct 30 07:52:13.965127 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:13.965072    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv" podStartSLOduration=1.965057751 podStartE2EDuration="1.965057751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:13.960587293 +0000 UTC m=+3294.393351185" watchObservedRunningTime="2026-10-30 07:52:13.965057751 +0000 UTC m=+3294.397821644"
Oct 30 07:52:14.932463 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:14.932427    6738 generic.go:356] "Generic (PLEG): container finished" podID="2bbed573-9d8d-4bfd-919d-4036986cc3b1" containerID="af1d2399eeefd61642b6cacfa4f06ea289d908a5f2ca6e36af4c695bd65a411a" exitCode=0
Oct 30 07:52:14.932909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:14.932515    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv" event={"ID":"2bbed573-9d8d-4bfd-919d-4036986cc3b1","Type":"ContainerDied","Data":"af1d2399eeefd61642b6cacfa4f06ea289d908a5f2ca6e36af4c695bd65a411a"}
Oct 30 07:52:15.141613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:15.141581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:16.117787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.117756    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:16.234139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.234047    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\" (UniqueName: \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\") pod \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") "
Oct 30 07:52:16.234139 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.234133    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\") pod \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") "
Oct 30 07:52:16.234477 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.234459    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\") pod \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\" (UID: \"2bbed573-9d8d-4bfd-919d-4036986cc3b1\") "
Oct 30 07:52:16.234537 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.234502    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca" pod "2bbed573-9d8d-4bfd-919d-4036986cc3b1" (UID: "2bbed573-9d8d-4bfd-919d-4036986cc3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:16.235104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.235082    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:16.236464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.236435    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq" pod "2bbed573-9d8d-4bfd-919d-4036986cc3b1" (UID: "2bbed573-9d8d-4bfd-919d-4036986cc3b1"). InnerVolumeSpecName "kube-api-access-hb9kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:16.236580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.236494    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca" pod "2bbed573-9d8d-4bfd-919d-4036986cc3b1" (UID: "2bbed573-9d8d-4bfd-919d-4036986cc3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:52:16.337172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.337133    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbed573-9d8d-4bfd-919d-4036986cc3b1-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:16.337172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.337170    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9kq\" (UniqueName: \"kubernetes.io/projected/2bbed573-9d8d-4bfd-919d-4036986cc3b1-kube-api-access-hb9kq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:16.951638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.951606    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv" event={"ID":"2bbed573-9d8d-4bfd-919d-4036986cc3b1","Type":"ContainerDied","Data":"d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0"}
Oct 30 07:52:16.951638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.951632    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"
Oct 30 07:52:16.951889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:16.951654    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60af5dda1d0adbc3dda9c8329583fae57e527d2c1d2d3e624c440119fb0bdd0"
Oct 30 07:52:17.038399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:17.038342    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:17.830438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:17.830403    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:17.953646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:17.953612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:18.147021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:18.146979    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:52:18.147243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:18.147209    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy" containerID="cri-o://1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d" gracePeriod=30
Oct 30 07:52:18.176193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:18.176158    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:19.031163 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:19.031124    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:19.191020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:19.190983    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:19.306770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:19.306669    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:19.435736 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:19.435687    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:20.940631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:20.940587    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:22.723249 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:22.723209    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:23.125744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.125713    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:23.259212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.259171    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:23.468608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.468574    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:52:23.587584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587551    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.587779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587593    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.587779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587619    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.587779 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587762    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.587931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587828    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.587931 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587874    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.588067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587942    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.588067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.587959    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.588067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588032    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.588246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588084    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.588246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588110    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.588246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588141    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\") pod \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\" (UID: \"a733a20c-bf1c-460b-86a6-2a849f7c17d8\") "
Oct 30 07:52:23.588246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588172    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.588531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588471    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:23.588959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588937    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.589067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588962    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.589067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588977    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.589067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.588991    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.589067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.589003    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.590159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.590125    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.590383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.590193    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:23.590383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.590216    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:23.590841 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.590823    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9" pod "a733a20c-bf1c-460b-86a6-2a849f7c17d8" (UID: "a733a20c-bf1c-460b-86a6-2a849f7c17d8"). InnerVolumeSpecName "kube-api-access-wscx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:23.703375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.690768    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.703652 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.703629    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.703777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.703758    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscx9\" (UniqueName: \"kubernetes.io/projected/a733a20c-bf1c-460b-86a6-2a849f7c17d8-kube-api-access-wscx9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.703878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:23.703865    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a733a20c-bf1c-460b-86a6-2a849f7c17d8-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:24.058682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.058594    6738 generic.go:356] "Generic (PLEG): container finished" podID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerID="1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d" exitCode=0
Oct 30 07:52:24.058682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.058649    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"
Oct 30 07:52:24.059325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.058676    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerDied","Data":"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"}
Oct 30 07:52:24.059325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.058743    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd" event={"ID":"a733a20c-bf1c-460b-86a6-2a849f7c17d8","Type":"ContainerDied","Data":"59a936243ef46e89a887ec4b8f5b13439e1178518dab0a2218de1d3acc2d828d"}
Oct 30 07:52:24.059325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.058777    6738 scope.go:122] "RemoveContainer" containerID="1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"
Oct 30 07:52:24.071398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.071377    6738 scope.go:122] "RemoveContainer" containerID="ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"
Oct 30 07:52:24.082778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.082758    6738 scope.go:122] "RemoveContainer" containerID="1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"
Oct 30 07:52:24.083066 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:24.083043    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d\": container with ID starting with 1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d not found: ID does not exist" containerID="1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"
Oct 30 07:52:24.083119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.083075    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d"} err="failed to get container status \"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d\": rpc error: code = NotFound desc = could not find container \"1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d\": container with ID starting with 1da42174fa3ea46f6e547285012f164a10a752cdd8cf254c16367eba9bb4938d not found: ID does not exist"
Oct 30 07:52:24.083119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.083092    6738 scope.go:122] "RemoveContainer" containerID="ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"
Oct 30 07:52:24.083352 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:24.083333    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4\": container with ID starting with ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4 not found: ID does not exist" containerID="ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"
Oct 30 07:52:24.083411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.083359    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4"} err="failed to get container status \"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4\": rpc error: code = NotFound desc = could not find container \"ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4\": container with ID starting with ebe2e8c4ffb9a0ba29f134a944de44e434e72ce1c314089ca2204f35ecfb5fb4 not found: ID does not exist"
Oct 30 07:52:24.114814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.114770    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:52:24.126845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.126813    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-blwvd"]
Oct 30 07:52:24.652659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:24.652627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:25.692572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:25.692536    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" path="/var/lib/kubelet/pods/a733a20c-bf1c-460b-86a6-2a849f7c17d8/volumes"
Oct 30 07:52:26.489875 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:26.489834    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:28.339614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:28.339574    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:30.310964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.307479    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:30.494936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.494901    6738 scope.go:122] "RemoveContainer" containerID="5342e8629a52975b3ac214ccaac38ae24f31fb782cd50391f78c711b26e8ca17"
Oct 30 07:52:30.508439 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.508418    6738 scope.go:122] "RemoveContainer" containerID="976adf9fa7b799b9337b596f9723bf7d44c8a060af1dda5f0e71c3a2ee611e10"
Oct 30 07:52:30.519361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.519338    6738 scope.go:122] "RemoveContainer" containerID="c4aa2262ea73d63de718a58f5d911d7c29074ee3843cf3389f4ca7c109fbd431"
Oct 30 07:52:30.529774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.529750    6738 scope.go:122] "RemoveContainer" containerID="599a00204087634dd8359fdabf8004999ba5b650c533cff4e3c9ddcdd18c3ef4"
Oct 30 07:52:30.539915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:30.539892    6738 scope.go:122] "RemoveContainer" containerID="89813a209e779e2ea086afde14a0527ea7061af4b077f2b0b446bd8210598cad"
Oct 30 07:52:32.264952 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:32.264910    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:33.879514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:33.879474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:34.428605 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:34.428567    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:35.847758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:35.847724    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:37.521309 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:37.521250    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:52:37.812183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:37.812095    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:39.628963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:39.628924    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:41.553900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:41.553862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:43.327135 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:43.327096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:45.341317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:45.339229    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:45.701008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:45.700972    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:52:45.701199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:45.701177    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy" containerID="cri-o://bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e" gracePeriod=30
Oct 30 07:52:47.362877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:47.362833    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:48.807627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:48.807591    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:52:48.808115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:48.807897    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy" containerID="cri-o://5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4" gracePeriod=30
Oct 30 07:52:49.275568 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:49.275526    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:51.002152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.002125    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:52:51.109579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109539    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109592    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109640    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109752    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109831    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109888    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.109998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.109941    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.110187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110012    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:51.110187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.110373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110133    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\") pod \"1bf68a9d-ba67-473e-a797-071eb232a1db\" (UID: \"1bf68a9d-ba67-473e-a797-071eb232a1db\") "
Oct 30 07:52:51.111222 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.111195    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.111467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110165    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:51.111467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110252    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:51.111467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.110592    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:51.111732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.111690    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:51.113026 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.112995    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:51.113218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.113197    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:51.113385 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.113365    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "kube-api-access-z7gmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:51.113637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.113615    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo" pod "1bf68a9d-ba67-473e-a797-071eb232a1db" (UID: "1bf68a9d-ba67-473e-a797-071eb232a1db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:51.213184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213150    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213178    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf68a9d-ba67-473e-a797-071eb232a1db-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213188    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213199    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213207    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213217    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf68a9d-ba67-473e-a797-071eb232a1db-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213227    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gmw\" (UniqueName: \"kubernetes.io/projected/1bf68a9d-ba67-473e-a797-071eb232a1db-kube-api-access-z7gmw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.213486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.213236    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf68a9d-ba67-473e-a797-071eb232a1db-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:51.255470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.255378    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:51.372225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.372183    6738 generic.go:356] "Generic (PLEG): container finished" podID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerID="bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e" exitCode=0
Oct 30 07:52:51.372464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.372241    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"
Oct 30 07:52:51.372464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.372234    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" event={"ID":"1bf68a9d-ba67-473e-a797-071eb232a1db","Type":"ContainerDied","Data":"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"}
Oct 30 07:52:51.372464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.372390    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t" event={"ID":"1bf68a9d-ba67-473e-a797-071eb232a1db","Type":"ContainerDied","Data":"c4e829f995262661591827f23b94a4bb9bc6025bd724d9ebc31c8b506ac885f9"}
Oct 30 07:52:51.372464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.372421    6738 scope.go:122] "RemoveContainer" containerID="bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"
Oct 30 07:52:51.399417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.399390    6738 scope.go:122] "RemoveContainer" containerID="bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"
Oct 30 07:52:51.399783 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:51.399757    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e\": container with ID starting with bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e not found: ID does not exist" containerID="bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"
Oct 30 07:52:51.399866 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.399792    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e"} err="failed to get container status \"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e\": rpc error: code = NotFound desc = could not find container \"bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e\": container with ID starting with bd19535ccf7e6bf95ed25bb36899038ec28375febe62bed117ddb8bb19286e5e not found: ID does not exist"
Oct 30 07:52:51.430532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.430485    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:52:51.435774 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.435745    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-w479t"]
Oct 30 07:52:51.696642 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:51.696604    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" path="/var/lib/kubelet/pods/1bf68a9d-ba67-473e-a797-071eb232a1db/volumes"
Oct 30 07:52:53.069741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:53.069704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:54.100910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.100882    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:52:54.161601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161497    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.161601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161553    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.161601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161610    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161674    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161722    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161850    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161895    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.161951    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.162145    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\") pod \"00511e0f-1eb2-4995-8455-b6618db0ed01\" (UID: \"00511e0f-1eb2-4995-8455-b6618db0ed01\") "
Oct 30 07:52:54.162634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.162527    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:54.162768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.162748    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:54.162926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.162911    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:54.163501 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.163462    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:54.163946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.163921    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:54.164700 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.164675    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.164858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.164842    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00511e0f-1eb2-4995-8455-b6618db0ed01-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.164972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.164958    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.165063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.165052    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.165177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.165148    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.166884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.165951    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:54.166884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.166238    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:54.167581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.167559    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:54.168141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.168106    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86" pod "00511e0f-1eb2-4995-8455-b6618db0ed01" (UID: "00511e0f-1eb2-4995-8455-b6618db0ed01"). InnerVolumeSpecName "kube-api-access-9gt86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:54.266695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.266658    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.266695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.266690    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.266695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.266699    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00511e0f-1eb2-4995-8455-b6618db0ed01-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.266944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.266708    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gt86\" (UniqueName: \"kubernetes.io/projected/00511e0f-1eb2-4995-8455-b6618db0ed01-kube-api-access-9gt86\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:52:54.400550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.400514    6738 generic.go:356] "Generic (PLEG): container finished" podID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerID="5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4" exitCode=0
Oct 30 07:52:54.400758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.400573    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"
Oct 30 07:52:54.400758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.400594    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerDied","Data":"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"}
Oct 30 07:52:54.400758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.400647    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m" event={"ID":"00511e0f-1eb2-4995-8455-b6618db0ed01","Type":"ContainerDied","Data":"bb8f9d3754270d49b970b3c1e808877d0a08bc966bed411ec5ff7d1ff136aa06"}
Oct 30 07:52:54.400758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.400676    6738 scope.go:122] "RemoveContainer" containerID="5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"
Oct 30 07:52:54.413303 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.413269    6738 scope.go:122] "RemoveContainer" containerID="cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"
Oct 30 07:52:54.424730 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.424707    6738 scope.go:122] "RemoveContainer" containerID="5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"
Oct 30 07:52:54.425067 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:54.425044    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4\": container with ID starting with 5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4 not found: ID does not exist" containerID="5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"
Oct 30 07:52:54.425169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.425080    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4"} err="failed to get container status \"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4\": rpc error: code = NotFound desc = could not find container \"5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4\": container with ID starting with 5a33dec218a92103cc762d688f8f5df64d6baa70bc971adf01ccc431634b8bb4 not found: ID does not exist"
Oct 30 07:52:54.425169 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.425106    6738 scope.go:122] "RemoveContainer" containerID="cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"
Oct 30 07:52:54.425415 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:52:54.425396    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552\": container with ID starting with cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552 not found: ID does not exist" containerID="cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"
Oct 30 07:52:54.425470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.425423    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552"} err="failed to get container status \"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552\": rpc error: code = NotFound desc = could not find container \"cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552\": container with ID starting with cae3c653b7621f7cef2538529dab467bd82b9230d75c10022a17b7bd68380552 not found: ID does not exist"
Oct 30 07:52:54.455661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.455623    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:52:54.468179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.468145    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d0c2f-openshift-default-f5cccd78f-8nt7m"]
Oct 30 07:52:54.834437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:54.834344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:55.692776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:55.692730    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" path="/var/lib/kubelet/pods/00511e0f-1eb2-4995-8455-b6618db0ed01/volumes"
Oct 30 07:52:56.778982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:56.778933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:58.692318 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:58.692266    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:59.005373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:59.005221    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:52:59.190402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:52:59.190349    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:00.900317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:00.873456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:03.057908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:03.057867    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:05.030412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:05.030367    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:07.096607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:07.096571    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:07.522799 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:53:07.522757    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:53:08.854865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:08.854831    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:10.913061 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:10.913026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:12.575661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:12.575624    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:13.345745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.345709    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:13.497527 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.497494    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"]
Oct 30 07:53:13.509551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.509514    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8fef73-pdxnv"]
Oct 30 07:53:13.547926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.547892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:13.693325 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.693280    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbed573-9d8d-4bfd-919d-4036986cc3b1" path="/var/lib/kubelet/pods/2bbed573-9d8d-4bfd-919d-4036986cc3b1/volumes"
Oct 30 07:53:13.707047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.707023    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:13.963119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:13.963031    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:15.711889 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:15.711845    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:17.831745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:17.831710    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:19.765000 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:19.764966    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:20.459213 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:20.459175    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:20.581366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:20.581330    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:20.856179 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:20.856095    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:21.449085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449049    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"]
Oct 30 07:53:21.449920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449903    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449921    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449944    6738 cpu_manager.go:414] "containerMap: removing container" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449951    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449960    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449965    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449979    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2bbed573-9d8d-4bfd-919d-4036986cc3b1" containerName="revoke-keys"
Oct 30 07:53:21.450003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.449985    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbed573-9d8d-4bfd-919d-4036986cc3b1" containerName="revoke-keys"
Oct 30 07:53:21.450264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.450132    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:21.450264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.450141    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf68a9d-ba67-473e-a797-071eb232a1db" containerName="istio-proxy"
Oct 30 07:53:21.450264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.450151    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbed573-9d8d-4bfd-919d-4036986cc3b1" containerName="revoke-keys"
Oct 30 07:53:21.450264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.450160    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:21.450264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.450171    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:21.453867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.453850    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.475398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.475364    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"]
Oct 30 07:53:21.480421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.479339    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.480421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.479453    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.480421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.479751    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mj9\" (UniqueName: \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.497371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.497277    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:53:21.497649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.497317    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:53:21.581191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.581147    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.581433 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.581400    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mj9\" (UniqueName: \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.581504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.581463    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.581813 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.581789    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.583895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.583870    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.605388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.605361    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mj9\" (UniqueName: \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\") pod \"maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.774343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.774239    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:21.843130 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:21.843092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:22.085618 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:53:22.085591    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cbb087_8a92_426f_a7bd_6b6220788913.slice/crio-d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff WatchSource:0}: Error finding container d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff: Status 404 returned error can't find the container with id d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff
Oct 30 07:53:22.085963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:22.085727    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"]
Oct 30 07:53:22.649923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:22.649823    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d" event={"ID":"78cbb087-8a92-426f-a7bd-6b6220788913","Type":"ContainerStarted","Data":"e930a88ec52476c6dfd1c5bb2c8a46086cd1bb7e5fc069a84ddab07a84faa1ea"}
Oct 30 07:53:22.649923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:22.649892    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d" event={"ID":"78cbb087-8a92-426f-a7bd-6b6220788913","Type":"ContainerStarted","Data":"d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff"}
Oct 30 07:53:22.692817 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:22.692764    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d" podStartSLOduration=1.692743186 podStartE2EDuration="1.692743186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:22.690192413 +0000 UTC m=+3363.122956306" watchObservedRunningTime="2026-10-30 07:53:22.692743186 +0000 UTC m=+3363.125514027"
Oct 30 07:53:23.661042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:23.660906    6738 generic.go:356] "Generic (PLEG): container finished" podID="78cbb087-8a92-426f-a7bd-6b6220788913" containerID="e930a88ec52476c6dfd1c5bb2c8a46086cd1bb7e5fc069a84ddab07a84faa1ea" exitCode=0
Oct 30 07:53:23.661548 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:23.661057    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d" event={"ID":"78cbb087-8a92-426f-a7bd-6b6220788913","Type":"ContainerDied","Data":"e930a88ec52476c6dfd1c5bb2c8a46086cd1bb7e5fc069a84ddab07a84faa1ea"}
Oct 30 07:53:23.734519 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:23.734455    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:24.844426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.844397    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:24.957476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.957394    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\") pod \"78cbb087-8a92-426f-a7bd-6b6220788913\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") "
Oct 30 07:53:24.957641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.957578    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\" (UniqueName: \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\") pod \"78cbb087-8a92-426f-a7bd-6b6220788913\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") "
Oct 30 07:53:24.957641 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.957637    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\") pod \"78cbb087-8a92-426f-a7bd-6b6220788913\" (UID: \"78cbb087-8a92-426f-a7bd-6b6220788913\") "
Oct 30 07:53:24.958134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.958075    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca" pod "78cbb087-8a92-426f-a7bd-6b6220788913" (UID: "78cbb087-8a92-426f-a7bd-6b6220788913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:53:24.958289 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.958241    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:24.959563 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.959533    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca" pod "78cbb087-8a92-426f-a7bd-6b6220788913" (UID: "78cbb087-8a92-426f-a7bd-6b6220788913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:53:24.959657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:24.959594    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9" pod "78cbb087-8a92-426f-a7bd-6b6220788913" (UID: "78cbb087-8a92-426f-a7bd-6b6220788913"). InnerVolumeSpecName "kube-api-access-p2mj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:25.060035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.059996    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbb087-8a92-426f-a7bd-6b6220788913-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:25.060035 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.060032    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mj9\" (UniqueName: \"kubernetes.io/projected/78cbb087-8a92-426f-a7bd-6b6220788913-kube-api-access-p2mj9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:25.679898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.679870    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"
Oct 30 07:53:25.679898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.679892    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d" event={"ID":"78cbb087-8a92-426f-a7bd-6b6220788913","Type":"ContainerDied","Data":"d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff"}
Oct 30 07:53:25.680178 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.679933    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48a8ff4b889cb42c873572f6eb324952278e68f91e3d3d36be406f2b47f2aff"
Oct 30 07:53:25.750945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:25.750892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:27.611376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:27.611336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:29.399895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:29.399860    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:31.352531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:31.352491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:32.125623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:32.125591    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:53:32.170756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:32.170718    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:53:32.332263 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:32.332219    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:32.489383 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:32.489262    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:33.833042 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:33.833002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:34.717177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:34.717143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:36.206260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:36.206225    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:37.126727 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.126690    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" podUID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" containerName="payload-processing" containerID="cri-o://e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74" gracePeriod=25
Oct 30 07:53:37.171539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.171474    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing" containerID="cri-o://1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b" gracePeriod=25
Oct 30 07:53:37.421087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.421052    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:53:37.526839 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:53:37.526800    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:53:37.579852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.579804    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\" (UniqueName: \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\") pod \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") "
Oct 30 07:53:37.580066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.579957    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\") pod \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\" (UID: \"5ab94f60-9bbc-4f4e-976b-58c9e401974e\") "
Oct 30 07:53:37.580453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.580427    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume" pod "5ab94f60-9bbc-4f4e-976b-58c9e401974e" (UID: "5ab94f60-9bbc-4f4e-976b-58c9e401974e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:53:37.580816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.580798    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab94f60-9bbc-4f4e-976b-58c9e401974e-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:37.582104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.582082    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl" pod "5ab94f60-9bbc-4f4e-976b-58c9e401974e" (UID: "5ab94f60-9bbc-4f4e-976b-58c9e401974e"). InnerVolumeSpecName "kube-api-access-gjngl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:37.682332 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.682236    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjngl\" (UniqueName: \"kubernetes.io/projected/5ab94f60-9bbc-4f4e-976b-58c9e401974e-kube-api-access-gjngl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:37.784394 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.784364    6738 generic.go:356] "Generic (PLEG): container finished" podID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" containerID="e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74" exitCode=0
Oct 30 07:53:37.784585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.784403    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"
Oct 30 07:53:37.784585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.784435    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" event={"ID":"5ab94f60-9bbc-4f4e-976b-58c9e401974e","Type":"ContainerDied","Data":"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"}
Oct 30 07:53:37.784585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.784478    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv" event={"ID":"5ab94f60-9bbc-4f4e-976b-58c9e401974e","Type":"ContainerDied","Data":"fe7cef8ad1c3bed839b49db744a4a08bf679e8f33622a1763c16ad21a3564750"}
Oct 30 07:53:37.784585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.784503    6738 scope.go:122] "RemoveContainer" containerID="e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"
Oct 30 07:53:37.798853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.798816    6738 scope.go:122] "RemoveContainer" containerID="e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"
Oct 30 07:53:37.799201 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:53:37.799176    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74\": container with ID starting with e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74 not found: ID does not exist" containerID="e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"
Oct 30 07:53:37.799342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.799208    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74"} err="failed to get container status \"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74\": rpc error: code = NotFound desc = could not find container \"e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74\": container with ID starting with e526f9e57911fc0ff9278b8623618f2dc2c1c0cc460e18a2f1f1d6e793c40f74 not found: ID does not exist"
Oct 30 07:53:37.845899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.845861    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:53:37.858190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:37.858157    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f480c6-85fb86dc8d-2rzrv"]
Oct 30 07:53:38.213916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:38.213878    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:39.692638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:39.692596    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" path="/var/lib/kubelet/pods/5ab94f60-9bbc-4f4e-976b-58c9e401974e/volumes"
Oct 30 07:53:39.695920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:39.695893    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:53:40.210176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:40.209049    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:42.221104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:42.221068    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:43.596246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.596208    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"]
Oct 30 07:53:43.597087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597071    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" containerName="payload-processing"
Oct 30 07:53:43.597157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597089    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" containerName="payload-processing"
Oct 30 07:53:43.597157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597132    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:43.597157 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597140    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a733a20c-bf1c-460b-86a6-2a849f7c17d8" containerName="istio-proxy"
Oct 30 07:53:43.597271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597159    6738 cpu_manager.go:414] "containerMap: removing container" podUID="78cbb087-8a92-426f-a7bd-6b6220788913" containerName="revoke-keys"
Oct 30 07:53:43.597271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597165    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cbb087-8a92-426f-a7bd-6b6220788913" containerName="revoke-keys"
Oct 30 07:53:43.597271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597177    6738 cpu_manager.go:414] "containerMap: removing container" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:43.597271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597181    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:43.597467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597311    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab94f60-9bbc-4f4e-976b-58c9e401974e" containerName="payload-processing"
Oct 30 07:53:43.597467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597330    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cbb087-8a92-426f-a7bd-6b6220788913" containerName="revoke-keys"
Oct 30 07:53:43.597467 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.597337    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="00511e0f-1eb2-4995-8455-b6618db0ed01" containerName="istio-proxy"
Oct 30 07:53:43.602544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.602527    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.620050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.620015    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"]
Oct 30 07:53:43.626232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.626208    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:53:43.626398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.626253    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:53:43.673215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.673177    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.673457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.673243    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.673457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.673397    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8w9v\" (UniqueName: \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.774827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.774787    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.775092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.774889    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.775092 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.775023    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8w9v\" (UniqueName: \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.776449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.776424    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.777878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.777854    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.799074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.799038    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8w9v\" (UniqueName: \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\") pod \"maas-api-revoke-sub-0f3a32a9-kmc6g\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:43.918414 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:43.918374    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:44.170059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:44.169965    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:44.258077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:44.258035    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"]
Oct 30 07:53:44.857050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:44.856961    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g" event={"ID":"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec","Type":"ContainerStarted","Data":"e1fce70610107e3a3ad8bd3db86f2a7031d84f571ef68e22e44b668d6528b6de"}
Oct 30 07:53:44.857050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:44.857022    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g" event={"ID":"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec","Type":"ContainerStarted","Data":"acef3280fc8ac341ed24279e42bba8af47ef95883b6f1a49954ba69f7e097a61"}
Oct 30 07:53:44.899742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:44.899680    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g" podStartSLOduration=1.899664268 podStartE2EDuration="1.899664268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:44.897661104 +0000 UTC m=+3385.330424997" watchObservedRunningTime="2026-10-30 07:53:44.899664268 +0000 UTC m=+3385.332428161"
Oct 30 07:53:45.866786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:45.866751    6738 generic.go:356] "Generic (PLEG): container finished" podID="cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" containerID="e1fce70610107e3a3ad8bd3db86f2a7031d84f571ef68e22e44b668d6528b6de" exitCode=0
Oct 30 07:53:45.867227 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:45.866836    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g" event={"ID":"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec","Type":"ContainerDied","Data":"e1fce70610107e3a3ad8bd3db86f2a7031d84f571ef68e22e44b668d6528b6de"}
Oct 30 07:53:45.938683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:45.938646    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:47.063597 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.063568    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:47.125539 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.125445    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\" (UniqueName: \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\") pod \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") "
Oct 30 07:53:47.125723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.125569    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\") pod \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") "
Oct 30 07:53:47.125723 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.125660    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\") pod \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\" (UID: \"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec\") "
Oct 30 07:53:47.126550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.126192    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca" pod "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" (UID: "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:53:47.126550 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.126516    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:47.127831 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.127806    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca" pod "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" (UID: "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:53:47.129072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.129046    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v" pod "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" (UID: "cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec"). InnerVolumeSpecName "kube-api-access-k8w9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:47.228235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.228194    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8w9v\" (UniqueName: \"kubernetes.io/projected/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-kube-api-access-k8w9v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:47.228235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.228229    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:47.777902 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.777866    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:47.889999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.889942    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g" event={"ID":"cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec","Type":"ContainerDied","Data":"acef3280fc8ac341ed24279e42bba8af47ef95883b6f1a49954ba69f7e097a61"}
Oct 30 07:53:47.890228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.890058    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acef3280fc8ac341ed24279e42bba8af47ef95883b6f1a49954ba69f7e097a61"
Oct 30 07:53:47.890228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:47.890155    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"
Oct 30 07:53:49.554978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:49.554941    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:49.696453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:49.696409    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:53:51.351535 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:51.351496    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:53.326020 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:53.325982    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:54.004944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:54.004909    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:55.283376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:55.283343    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:57.194963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.194926    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:57.459243 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.459215    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:53:57.515560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.515527    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\") pod \"4ed16e67-8162-4070-acdf-ce03d7865879\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") "
Oct 30 07:53:57.515743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.515574    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\" (UniqueName: \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\") pod \"4ed16e67-8162-4070-acdf-ce03d7865879\" (UID: \"4ed16e67-8162-4070-acdf-ce03d7865879\") "
Oct 30 07:53:57.516044 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.516015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume" pod "4ed16e67-8162-4070-acdf-ce03d7865879" (UID: "4ed16e67-8162-4070-acdf-ce03d7865879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:53:57.516386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.516365    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed16e67-8162-4070-acdf-ce03d7865879-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:57.517826 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.517792    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw" pod "4ed16e67-8162-4070-acdf-ce03d7865879" (UID: "4ed16e67-8162-4070-acdf-ce03d7865879"). InnerVolumeSpecName "kube-api-access-hjgsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:57.619746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.619704    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjgsw\" (UniqueName: \"kubernetes.io/projected/4ed16e67-8162-4070-acdf-ce03d7865879-kube-api-access-hjgsw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:53:57.982105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.982014    6738 generic.go:356] "Generic (PLEG): container finished" podID="4ed16e67-8162-4070-acdf-ce03d7865879" containerID="1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b" exitCode=0
Oct 30 07:53:57.982105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.982091    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" event={"ID":"4ed16e67-8162-4070-acdf-ce03d7865879","Type":"ContainerDied","Data":"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"}
Oct 30 07:53:57.982369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.982100    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"
Oct 30 07:53:57.982369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.982136    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8" event={"ID":"4ed16e67-8162-4070-acdf-ce03d7865879","Type":"ContainerDied","Data":"11daa5805ca61a41b2ba5ed47ee237e71c4e342f8684b114f2801ca69eb0272e"}
Oct 30 07:53:57.982369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.982159    6738 scope.go:122] "RemoveContainer" containerID="1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"
Oct 30 07:53:57.993784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.993763    6738 scope.go:122] "RemoveContainer" containerID="1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"
Oct 30 07:53:57.994086 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:53:57.994049    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b\": container with ID starting with 1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b not found: ID does not exist" containerID="1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"
Oct 30 07:53:57.994199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:57.994088    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b"} err="failed to get container status \"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b\": rpc error: code = NotFound desc = could not find container \"1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b\": container with ID starting with 1d5ddacba7d21bf1733ec024822d44e64d0e7e2bfac72aefc83c80c7c38aa85b not found: ID does not exist"
Oct 30 07:53:58.037503 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.037469    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:53:58.050111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.050075    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f480c6-7fc897d959-c7gf8"]
Oct 30 07:53:58.366361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.366247    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"]
Oct 30 07:53:58.367085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367069    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" containerName="revoke-keys"
Oct 30 07:53:58.367145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367086    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" containerName="revoke-keys"
Oct 30 07:53:58.367145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367106    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing"
Oct 30 07:53:58.367145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367111    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing"
Oct 30 07:53:58.367259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367241    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" containerName="revoke-keys"
Oct 30 07:53:58.367259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.367251    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" containerName="payload-pre-processing"
Oct 30 07:53:58.372456 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.372438    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.388058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.388030    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:53:58.388058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.388030    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:53:58.389800 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.389771    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"]
Oct 30 07:53:58.432066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.432020    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.432316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.432169    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.432316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.432237    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7xk\" (UniqueName: \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.534002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.533956    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.534201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.534058    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.534201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.534086    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7xk\" (UniqueName: \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.534801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.534770    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.536428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.536408    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.558442 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.558414    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7xk\" (UniqueName: \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\") pod \"maas-api-revoke-sub-a1f5bb69-kk7jv\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:58.688504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:58.688467    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:53:59.000821 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:53:59.000783    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004e9b0_6808_41d1_a132_783dd254a6f5.slice/crio-728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921 WatchSource:0}: Error finding container 728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921: Status 404 returned error can't find the container with id 728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921
Oct 30 07:53:59.001908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:59.001884    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"]
Oct 30 07:53:59.030151 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:59.030122    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:53:59.692237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:53:59.692153    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed16e67-8162-4070-acdf-ce03d7865879" path="/var/lib/kubelet/pods/4ed16e67-8162-4070-acdf-ce03d7865879/volumes"
Oct 30 07:54:00.000953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:00.000917    6738 generic.go:356] "Generic (PLEG): container finished" podID="f004e9b0-6808-41d1-a132-783dd254a6f5" containerID="c498f24aba63068bbb44faaa7300b00e68448fd04a73c5a54d15a61b8c206e21" exitCode=0
Oct 30 07:54:00.001118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:00.001006    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv" event={"ID":"f004e9b0-6808-41d1-a132-783dd254a6f5","Type":"ContainerDied","Data":"c498f24aba63068bbb44faaa7300b00e68448fd04a73c5a54d15a61b8c206e21"}
Oct 30 07:54:00.001118 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:00.001059    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv" event={"ID":"f004e9b0-6808-41d1-a132-783dd254a6f5","Type":"ContainerStarted","Data":"728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921"}
Oct 30 07:54:00.318031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:00.317942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:00.842344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:00.842280    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:01.212034 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.212003    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:54:01.276626 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.276590    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\" (UniqueName: \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\") pod \"f004e9b0-6808-41d1-a132-783dd254a6f5\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") "
Oct 30 07:54:01.276846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.276683    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\") pod \"f004e9b0-6808-41d1-a132-783dd254a6f5\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") "
Oct 30 07:54:01.276846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.276775    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\") pod \"f004e9b0-6808-41d1-a132-783dd254a6f5\" (UID: \"f004e9b0-6808-41d1-a132-783dd254a6f5\") "
Oct 30 07:54:01.277171 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.277140    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca" pod "f004e9b0-6808-41d1-a132-783dd254a6f5" (UID: "f004e9b0-6808-41d1-a132-783dd254a6f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:01.278112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.278083    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:01.279710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.279681    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk" pod "f004e9b0-6808-41d1-a132-783dd254a6f5" (UID: "f004e9b0-6808-41d1-a132-783dd254a6f5"). InnerVolumeSpecName "kube-api-access-pc7xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:01.279927 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.279901    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca" pod "f004e9b0-6808-41d1-a132-783dd254a6f5" (UID: "f004e9b0-6808-41d1-a132-783dd254a6f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:54:01.379353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.379304    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc7xk\" (UniqueName: \"kubernetes.io/projected/f004e9b0-6808-41d1-a132-783dd254a6f5-kube-api-access-pc7xk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:01.379353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:01.379342    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004e9b0-6808-41d1-a132-783dd254a6f5-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:02.019739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:02.019652    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv" event={"ID":"f004e9b0-6808-41d1-a132-783dd254a6f5","Type":"ContainerDied","Data":"728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921"}
Oct 30 07:54:02.019739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:02.019694    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"
Oct 30 07:54:02.019739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:02.019703    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="728db07025020ac8e3a6d46b7b852ddc143a2c3821cf626ffa3ceaf86a207921"
Oct 30 07:54:02.749827 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:02.749786    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:04.543493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:04.543462    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:06.456466 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:06.456430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:07.520316 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:54:07.520262    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:54:08.353796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:08.353760    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:09.376398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.376364    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:09.926977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.926936    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"]
Oct 30 07:54:09.927816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.927798    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f004e9b0-6808-41d1-a132-783dd254a6f5" containerName="revoke-keys"
Oct 30 07:54:09.927885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.927817    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004e9b0-6808-41d1-a132-783dd254a6f5" containerName="revoke-keys"
Oct 30 07:54:09.927962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.927953    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004e9b0-6808-41d1-a132-783dd254a6f5" containerName="revoke-keys"
Oct 30 07:54:09.932122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.932101    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:09.948917 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.948887    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"]
Oct 30 07:54:09.955776 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.955742    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:54:09.955945 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.955846    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:54:09.965016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.964980    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:09.965247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.965226    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:09.965361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:09.965273    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlpb\" (UniqueName: \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.066553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.066508    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.066785 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.066566    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlpb\" (UniqueName: \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.066864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.066778    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.067180 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.067161    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.069185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.069165    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.090577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.090547    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlpb\" (UniqueName: \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\") pod \"maas-api-revoke-sub-56bf9076-zdngt\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.238879 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.238795    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:10.248959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.248926    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:10.611748 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:54:10.611716    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e29a031_3c6f_4952_a7e2_32bd8090a0ec.slice/crio-d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b WatchSource:0}: Error finding container d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b: Status 404 returned error can't find the container with id d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b
Oct 30 07:54:10.612212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:10.612036    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"]
Oct 30 07:54:11.100962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:11.100927    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt" event={"ID":"1e29a031-3c6f-4952-a7e2-32bd8090a0ec","Type":"ContainerStarted","Data":"91dbda671dfb7b373d186076529339e0c072611b6bfbe779052e6eb33fa0e367"}
Oct 30 07:54:11.101158 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:11.100982    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt" event={"ID":"1e29a031-3c6f-4952-a7e2-32bd8090a0ec","Type":"ContainerStarted","Data":"d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b"}
Oct 30 07:54:11.155878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:11.155820    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt" podStartSLOduration=2.155806797 podStartE2EDuration="2.155806797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:11.152656338 +0000 UTC m=+3411.585420232" watchObservedRunningTime="2026-10-30 07:54:11.155806797 +0000 UTC m=+3411.588570690"
Oct 30 07:54:12.110858 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:12.110822    6738 generic.go:356] "Generic (PLEG): container finished" podID="1e29a031-3c6f-4952-a7e2-32bd8090a0ec" containerID="91dbda671dfb7b373d186076529339e0c072611b6bfbe779052e6eb33fa0e367" exitCode=0
Oct 30 07:54:12.111285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:12.110894    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt" event={"ID":"1e29a031-3c6f-4952-a7e2-32bd8090a0ec","Type":"ContainerDied","Data":"91dbda671dfb7b373d186076529339e0c072611b6bfbe779052e6eb33fa0e367"}
Oct 30 07:54:12.149483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:12.149440    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:13.282141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.282113    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:13.438688 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.438657    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\") pod \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") "
Oct 30 07:54:13.438904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.438724    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\" (UniqueName: \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\") pod \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") "
Oct 30 07:54:13.438904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.438759    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\") pod \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\" (UID: \"1e29a031-3c6f-4952-a7e2-32bd8090a0ec\") "
Oct 30 07:54:13.439111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.439083    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca" pod "1e29a031-3c6f-4952-a7e2-32bd8090a0ec" (UID: "1e29a031-3c6f-4952-a7e2-32bd8090a0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:13.439360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.439344    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:13.441001 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.440975    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca" pod "1e29a031-3c6f-4952-a7e2-32bd8090a0ec" (UID: "1e29a031-3c6f-4952-a7e2-32bd8090a0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:54:13.441137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.441015    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb" pod "1e29a031-3c6f-4952-a7e2-32bd8090a0ec" (UID: "1e29a031-3c6f-4952-a7e2-32bd8090a0ec"). InnerVolumeSpecName "kube-api-access-8tlpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:13.541410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.541374    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tlpb\" (UniqueName: \"kubernetes.io/projected/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-kube-api-access-8tlpb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:13.541410 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.541404    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e29a031-3c6f-4952-a7e2-32bd8090a0ec-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:13.919058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:13.919024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:14.128421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:14.128395    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"
Oct 30 07:54:14.128421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:14.128413    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt" event={"ID":"1e29a031-3c6f-4952-a7e2-32bd8090a0ec","Type":"ContainerDied","Data":"d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b"}
Oct 30 07:54:14.128645 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:14.128455    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d926effb4a0f9a3df809a6a30a6268e7128fbed0dc87c8a9da0ad48c6b35a09b"
Oct 30 07:54:15.723129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:15.723099    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:17.728011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:17.727907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:19.754126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:19.754092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:21.656330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:21.656282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:23.203936 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:23.203892    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:23.924458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:23.924415    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:24.145543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.145504    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"]
Oct 30 07:54:24.216377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.215363    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:24.216377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.216223    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f480c6-mkm4d"]
Oct 30 07:54:24.383424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.383383    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"]
Oct 30 07:54:24.384256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.384237    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1e29a031-3c6f-4952-a7e2-32bd8090a0ec" containerName="revoke-keys"
Oct 30 07:54:24.384372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.384258    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e29a031-3c6f-4952-a7e2-32bd8090a0ec" containerName="revoke-keys"
Oct 30 07:54:24.384423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.384413    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e29a031-3c6f-4952-a7e2-32bd8090a0ec" containerName="revoke-keys"
Oct 30 07:54:24.386809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.386779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:24.388365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.388349    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.408287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.408252    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"]
Oct 30 07:54:24.416823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.416793    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:54:24.417174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.417127    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:54:24.491659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.491562    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.491881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.491860    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lr6\" (UniqueName: \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.491954 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.491895    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.593823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.593786    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lr6\" (UniqueName: \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.594059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.593831    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.594059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.593961    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.594594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.594571    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.596514 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.596483    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.626377 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.626344    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lr6\" (UniqueName: \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\") pod \"maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:24.645269 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.645235    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:24.704314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:24.704273    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:25.020261 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:54:25.020174    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cab7d73_6354_4215_82f7_a058db985438.slice/crio-d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a WatchSource:0}: Error finding container d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a: Status 404 returned error can't find the container with id d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a
Oct 30 07:54:25.020697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:25.020586    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"]
Oct 30 07:54:25.255321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:25.255253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b" event={"ID":"5cab7d73-6354-4215-82f7-a058db985438","Type":"ContainerStarted","Data":"d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a"}
Oct 30 07:54:25.693603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:25.693514    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cbb087-8a92-426f-a7bd-6b6220788913" path="/var/lib/kubelet/pods/78cbb087-8a92-426f-a7bd-6b6220788913/volumes"
Oct 30 07:54:26.265638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:26.265604    6738 generic.go:356] "Generic (PLEG): container finished" podID="5cab7d73-6354-4215-82f7-a058db985438" containerID="a0cdae491c4de8ecad1319cac5c7e258023aad3f96dff5fe131f5718dfa943d6" exitCode=0
Oct 30 07:54:26.266125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:26.265684    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b" event={"ID":"5cab7d73-6354-4215-82f7-a058db985438","Type":"ContainerDied","Data":"a0cdae491c4de8ecad1319cac5c7e258023aad3f96dff5fe131f5718dfa943d6"}
Oct 30 07:54:26.315267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:26.315225    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:27.438095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.438069    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:27.535818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.535726    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\") pod \"5cab7d73-6354-4215-82f7-a058db985438\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") "
Oct 30 07:54:27.536014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.535832    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\" (UniqueName: \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\") pod \"5cab7d73-6354-4215-82f7-a058db985438\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") "
Oct 30 07:54:27.536075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.536024    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\") pod \"5cab7d73-6354-4215-82f7-a058db985438\" (UID: \"5cab7d73-6354-4215-82f7-a058db985438\") "
Oct 30 07:54:27.536464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.536431    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca" pod "5cab7d73-6354-4215-82f7-a058db985438" (UID: "5cab7d73-6354-4215-82f7-a058db985438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:27.536683 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.536663    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cab7d73-6354-4215-82f7-a058db985438-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:27.537972 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.537950    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca" pod "5cab7d73-6354-4215-82f7-a058db985438" (UID: "5cab7d73-6354-4215-82f7-a058db985438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:54:27.538049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.538025    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6" pod "5cab7d73-6354-4215-82f7-a058db985438" (UID: "5cab7d73-6354-4215-82f7-a058db985438"). InnerVolumeSpecName "kube-api-access-62lr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:27.640814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.640217    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cab7d73-6354-4215-82f7-a058db985438-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:27.640814 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:27.640274    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62lr6\" (UniqueName: \"kubernetes.io/projected/5cab7d73-6354-4215-82f7-a058db985438-kube-api-access-62lr6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:28.165218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:28.165176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:28.283134 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:28.283103    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"
Oct 30 07:54:28.283338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:28.283102    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b" event={"ID":"5cab7d73-6354-4215-82f7-a058db985438","Type":"ContainerDied","Data":"d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a"}
Oct 30 07:54:28.283338 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:28.283236    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d20b36ba2b425881940b64d4d3c6c5c4d215898c08b39c3a6023a5f782a6575a"
Oct 30 07:54:29.271742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:29.271662    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:54:29.272206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:29.271865    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" containerID="cri-o://5f62509eec1beea140e1f1c6770a2f52f3a1e7e775dc9199903dd22dba35f1c2" gracePeriod=30
Oct 30 07:54:30.016978 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:30.016945    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:31.074705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.074621    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 07:54:31.075573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.075549    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5cab7d73-6354-4215-82f7-a058db985438" containerName="revoke-keys"
Oct 30 07:54:31.075573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.075572    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cab7d73-6354-4215-82f7-a058db985438" containerName="revoke-keys"
Oct 30 07:54:31.075791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.075779    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cab7d73-6354-4215-82f7-a058db985438" containerName="revoke-keys"
Oct 30 07:54:31.081040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.081021    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.119051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.119016    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 07:54:31.170282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.170250    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:31.204744 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.204706    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657gk\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.204967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.204786    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.204967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.204868    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.204967 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.204920    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.205152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.205004    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.205355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.205275    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.205450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.205349    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.205510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.205486    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.205564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.205512    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307325    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307371 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307376    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307415    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307433    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307511    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657gk\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307588    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.307656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307617    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308003    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308260 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308234    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308324    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308043    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308441 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.307853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308654    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.308805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.308756    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.310230 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.310207    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.310336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.310253    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.336789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.336704    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657gk\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.337064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.337044    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skjxh\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.351974 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.351934    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:31.407369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.407282    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:31.495177 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.495128    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:54:31.495795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.495732    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" containerID="cri-o://925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080" gracePeriod=30
Oct 30 07:54:31.524348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.521544    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:31.804316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.803609    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:54:31.804316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.803671    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:54:31.853037 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:54:31.853010    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb30c606_5ce5_4aa6_9225_cbd6fe5af60c.slice/crio-afe3f054388ddc43bdb01d4b5112d3ef1eff041e52a3906af76e6f26f8d9c381 WatchSource:0}: Error finding container afe3f054388ddc43bdb01d4b5112d3ef1eff041e52a3906af76e6f26f8d9c381: Status 404 returned error can't find the container with id afe3f054388ddc43bdb01d4b5112d3ef1eff041e52a3906af76e6f26f8d9c381
Oct 30 07:54:31.854719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.854698    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 07:54:31.856573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.856539    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:54:31.856651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.856610    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:54:31.856651 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:31.856641    6738 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 07:54:32.113003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.112907    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:32.323355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.323311    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" event={"ID":"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c","Type":"ContainerStarted","Data":"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"}
Oct 30 07:54:32.323543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.323387    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" event={"ID":"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c","Type":"ContainerStarted","Data":"afe3f054388ddc43bdb01d4b5112d3ef1eff041e52a3906af76e6f26f8d9c381"}
Oct 30 07:54:32.387360 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.387286    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podStartSLOduration=1.3872655950000001 podStartE2EDuration="1.387265595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:32.382101338 +0000 UTC m=+3432.814865263" watchObservedRunningTime="2026-10-30 07:54:32.387265595 +0000 UTC m=+3432.820029493"
Oct 30 07:54:32.408088 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.408054    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:32.410072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.410039    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:32.410244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.410089    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:32.434256 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.434215    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:32.601588 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.601553    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:32.733876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:32.733762    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:33.308095 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:33.308053    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:33.408491 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:33.408457    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:33.408687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:33.408509    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:34.140878 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.140844    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:34.341330 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.341268    6738 generic.go:356] "Generic (PLEG): container finished" podID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerID="5f62509eec1beea140e1f1c6770a2f52f3a1e7e775dc9199903dd22dba35f1c2" exitCode=0
Oct 30 07:54:34.341724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.341351    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" event={"ID":"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75","Type":"ContainerDied","Data":"5f62509eec1beea140e1f1c6770a2f52f3a1e7e775dc9199903dd22dba35f1c2"}
Oct 30 07:54:34.408870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.408144    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:34.408870 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.408210    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:34.538423 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.538392    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:54:34.566627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566597    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566669    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566771    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566846    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566864    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566891    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566910    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.566968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566929    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:34.567255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.566936    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.567255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.567033    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:34.567255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.567040    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\") pod \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\" (UID: \"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75\") "
Oct 30 07:54:34.567255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.567199    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:34.567534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.567267    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:34.595253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.568024    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.595253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.568053    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.595253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.568067    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.595253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.568082    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.595253 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.590588    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:34.595737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.595347    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:54:34.595737 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.595407    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:34.596704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.596513    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "kube-api-access-vjfgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:34.601275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.599535    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token" pod "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" (UID: "3b783cfe-1e62-4828-81cd-2c1f4bb8ae75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:34.684376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.684118    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.684376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.684152    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.684376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.684169    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfgg\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-kube-api-access-vjfgg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.684376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.684183    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.684376 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.684198    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:34.702086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.702049    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:34.821996 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.821961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:34.938787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:34.938699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:35.027242 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.027212    6738 patch_prober.go:29] interesting pod/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:54:35.027440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.027262    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:54:35.419320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.416324    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:35.419320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.416394    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:35.463316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.455422    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8" event={"ID":"3b783cfe-1e62-4828-81cd-2c1f4bb8ae75","Type":"ContainerDied","Data":"f063e9eb6dfab6db42cf6c8020da57e8d245af622a85be710768387efa376a15"}
Oct 30 07:54:35.463316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.455573    6738 scope.go:122] "RemoveContainer" containerID="5f62509eec1beea140e1f1c6770a2f52f3a1e7e775dc9199903dd22dba35f1c2"
Oct 30 07:54:35.463316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.455797    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"
Oct 30 07:54:35.598322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.597807    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:54:35.602547 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.602515    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-xstc8"]
Oct 30 07:54:35.701739 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:35.701668    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" path="/var/lib/kubelet/pods/3b783cfe-1e62-4828-81cd-2c1f4bb8ae75/volumes"
Oct 30 07:54:36.233181 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:36.233141    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:36.408373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:36.408341    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:36.408656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:36.408623    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:37.240735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.240707    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:54:37.352823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.352712    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.352823 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.352784    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.352857    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.352922    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353021    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353132 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353038    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353175    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353204    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353232    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\") pod \"2a1b3a58-f2b6-4529-9989-234cfab4f441\" (UID: \"2a1b3a58-f2b6-4529-9989-234cfab4f441\") "
Oct 30 07:54:37.353761 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353729    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:37.354225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.353988    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.354225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.354117    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1b3a58-f2b6-4529-9989-234cfab4f441-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.354225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.354141    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.354225 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.354194    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.355065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.354978    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.355065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.355040    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.357398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.356589    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.357398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.356743    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "kube-api-access-gbhqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:37.357398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.356820    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:37.358238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.358218    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo" pod "2a1b3a58-f2b6-4529-9989-234cfab4f441" (UID: "2a1b3a58-f2b6-4529-9989-234cfab4f441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:54:37.408366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.408330    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:37.408566 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.408386    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:37.455793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455758    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.455793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455789    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.455793 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455800    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.456029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455810    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.456029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455820    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.456029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455833    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhqx\" (UniqueName: \"kubernetes.io/projected/2a1b3a58-f2b6-4529-9989-234cfab4f441-kube-api-access-gbhqx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.456029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.455842    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1b3a58-f2b6-4529-9989-234cfab4f441-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.486047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.486007    6738 generic.go:356] "Generic (PLEG): container finished" podID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerID="925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080" exitCode=0
Oct 30 07:54:37.486399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.486369    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerDied","Data":"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"}
Oct 30 07:54:37.486624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.486603    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s" event={"ID":"2a1b3a58-f2b6-4529-9989-234cfab4f441","Type":"ContainerDied","Data":"86d5b2f4a56b5f173e2fe8345c9b10a7acc7e35a87d5845844a562567007e51f"}
Oct 30 07:54:37.488047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.486825    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"
Oct 30 07:54:37.488421 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.487864    6738 scope.go:122] "RemoveContainer" containerID="925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"
Oct 30 07:54:37.512054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.512033    6738 scope.go:122] "RemoveContainer" containerID="94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"
Oct 30 07:54:37.526587 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.526560    6738 scope.go:122] "RemoveContainer" containerID="925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"
Oct 30 07:54:37.527348 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:54:37.527279    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080\": container with ID starting with 925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080 not found: ID does not exist" containerID="925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"
Oct 30 07:54:37.527453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.527359    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080"} err="failed to get container status \"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080\": rpc error: code = NotFound desc = could not find container \"925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080\": container with ID starting with 925d9030dbb055b4ed87c07addb19acd97ffb9d2acea0306446d77766a24e080 not found: ID does not exist"
Oct 30 07:54:37.527453 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.527378    6738 scope.go:122] "RemoveContainer" containerID="94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"
Oct 30 07:54:37.527707 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:54:37.527677    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941\": container with ID starting with 94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941 not found: ID does not exist" containerID="94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"
Oct 30 07:54:37.527770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.527719    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941"} err="failed to get container status \"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941\": rpc error: code = NotFound desc = could not find container \"94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941\": container with ID starting with 94af84d684fbf8e9add80569282cbb0968a120e1d8ba02e7bdda5d35f5dc3941 not found: ID does not exist"
Oct 30 07:54:37.533153 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:54:37.533125    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:54:37.545838 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.545812    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:54:37.551545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.551519    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f480c6-openshift-default-6b76996fd7-jk78s"]
Oct 30 07:54:37.699818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:37.699786    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" path="/var/lib/kubelet/pods/2a1b3a58-f2b6-4529-9989-234cfab4f441/volumes"
Oct 30 07:54:38.329443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:38.329407    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:38.417317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:38.414517    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:38.417317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:38.414578    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:39.408654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:39.408612    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:39.409093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:39.408684    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:40.207129 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:40.207088    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:40.408722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:40.408682    6738 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused" start-of-body=
Oct 30 07:54:40.409206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:40.408771    6738 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.164:15021/healthz/ready\": dial tcp 10.133.1.164:15021: connect: connection refused"
Oct 30 07:54:41.762067 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:41.762027    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:41.762614 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:41.762430    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:41.763579 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:41.763553    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 07:54:42.041426 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:42.041279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:43.856509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:43.856461    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:44.181677 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:44.181644    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:44.296428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:44.296384    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:45.778175 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:45.778146    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:47.031634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:47.031598    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"]
Oct 30 07:54:47.043637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:47.043609    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a4094-jt4hk"]
Oct 30 07:54:47.692531 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:47.692498    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8281e1-8447-42a0-9261-f96c52c71e6d" path="/var/lib/kubelet/pods/2c8281e1-8447-42a0-9261-f96c52c71e6d/volumes"
Oct 30 07:54:47.815411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:47.815375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:49.663274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:49.663239    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:51.505956 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:51.505911    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:53.316901 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:53.316862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:55.260375 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:55.260340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:55.648862 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:55.648823    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:57.140946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:57.140908    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:58.976329 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:58.973028    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:54:59.635245 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:54:59.635209    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:00.032405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:00.032321    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"]
Oct 30 07:55:00.045287 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:00.045250    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19309ad2-bnggm"]
Oct 30 07:55:00.830980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:00.830940    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:01.697204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:01.697158    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6777eab-f983-48d8-941f-29877ad83b42" path="/var/lib/kubelet/pods/c6777eab-f983-48d8-941f-29877ad83b42/volumes"
Oct 30 07:55:02.704554 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:02.704524    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:04.532017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:04.531980    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:06.104721 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:06.104670    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:06.455116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:06.455072    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:07.524274 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:55:07.524233    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:55:08.346482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:08.346443    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:10.184188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:10.184156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:11.948236 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:11.948189    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:13.836411 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:13.836377    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:15.745820 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:15.745784    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:17.552655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:17.552613    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:19.348743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:19.348704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:21.031493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.031454    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:21.546316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.546260    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:21.710450 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.710416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:21.759340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.759305    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"]
Oct 30 07:55:21.792002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.791961    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-91c8c2-9fl2b"]
Oct 30 07:55:21.825824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:21.825738    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:22.283204 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:22.283172    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:23.692976 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:23.692942    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cab7d73-6354-4215-82f7-a058db985438" path="/var/lib/kubelet/pods/5cab7d73-6354-4215-82f7-a058db985438/volumes"
Oct 30 07:55:23.720468 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:23.720422    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:24.154772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:24.154734    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:24.272228 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:24.272196    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:25.578406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:25.578368    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:27.525604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:27.522956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:28.812565 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:28.812527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:28.942881 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:28.942842    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:29.185212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.185080    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:29.321217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.321180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:29.794867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.794828    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"]
Oct 30 07:55:29.795788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.795763    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:29.795788 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.795787    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:29.796443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.795856    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy"
Oct 30 07:55:29.796443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.795865    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy"
Oct 30 07:55:29.796443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.796084    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:29.796443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.796113    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b783cfe-1e62-4828-81cd-2c1f4bb8ae75" containerName="istio-proxy"
Oct 30 07:55:29.800226 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.800206    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.816926 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.816895    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"]
Oct 30 07:55:29.824384 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.824355    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:55:29.824623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.824359    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:55:29.869167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.869126    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.869402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.869212    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jdz\" (UniqueName: \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.869489 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.869396    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.971310 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.971264    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.971499 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.971485    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.971551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.971530    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jdz\" (UniqueName: \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.972133 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.972105    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.973801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.973779    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:29.995512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:29.995482    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jdz\" (UniqueName: \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\") pod \"maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:30.121479 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:30.121424    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:30.433205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:30.433177    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"]
Oct 30 07:55:30.434169 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:55:30.434129    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6a397c_526c_40be_9a53_1e1fd17bcff6.slice/crio-9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b WatchSource:0}: Error finding container 9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b: Status 404 returned error can't find the container with id 9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b
Oct 30 07:55:30.967428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:30.967328    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk" event={"ID":"3a6a397c-526c-40be-9a53-1e1fd17bcff6","Type":"ContainerStarted","Data":"95177c693fe43d5ad617ac45df85d44e4a406109619eab9dc2095838b6274d48"}
Oct 30 07:55:30.967428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:30.967397    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk" event={"ID":"3a6a397c-526c-40be-9a53-1e1fd17bcff6","Type":"ContainerStarted","Data":"9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b"}
Oct 30 07:55:31.014153 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:31.014095    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk" podStartSLOduration=2.014080739 podStartE2EDuration="2.014080739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:55:31.009485461 +0000 UTC m=+3491.442249354" watchObservedRunningTime="2026-10-30 07:55:31.014080739 +0000 UTC m=+3491.446844632"
Oct 30 07:55:31.358916 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:31.358832    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:31.976900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:31.976859    6738 generic.go:356] "Generic (PLEG): container finished" podID="3a6a397c-526c-40be-9a53-1e1fd17bcff6" containerID="95177c693fe43d5ad617ac45df85d44e4a406109619eab9dc2095838b6274d48" exitCode=0
Oct 30 07:55:31.977359 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:31.976947    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk" event={"ID":"3a6a397c-526c-40be-9a53-1e1fd17bcff6","Type":"ContainerDied","Data":"95177c693fe43d5ad617ac45df85d44e4a406109619eab9dc2095838b6274d48"}
Oct 30 07:55:32.065013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.064978    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"]
Oct 30 07:55:32.065865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.065848    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:32.065944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.065866    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:32.066029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.066002    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b3a58-f2b6-4529-9989-234cfab4f441" containerName="istio-proxy"
Oct 30 07:55:32.070049 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.070024    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.086741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.086710    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"]
Oct 30 07:55:32.220711 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.220675    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.220904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.220869    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.220985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.220968    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94slc\" (UniqueName: \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.321989    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.322058    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94slc\" (UniqueName: \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.322266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.322197    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.322780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.322758    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.324617 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.324596    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.348185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.348148    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94slc\" (UniqueName: \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\") pod \"maas-api-revoke-sub-6a0c763c-jchw4\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.388145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.388112    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:32.698475 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:55:32.698441    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef2cd1a_a06d_4b82_8c7b_e7b4de1ddd30.slice/crio-9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30 WatchSource:0}: Error finding container 9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30: Status 404 returned error can't find the container with id 9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30
Oct 30 07:55:32.698704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.698681    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"]
Oct 30 07:55:32.987250 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:32.987161    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4" event={"ID":"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30","Type":"ContainerStarted","Data":"9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30"}
Oct 30 07:55:33.136205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.136179    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:33.142703 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.142669    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:33.235775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.235712    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\") pod \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") "
Oct 30 07:55:33.236231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.235892    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\" (UniqueName: \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\") pod \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") "
Oct 30 07:55:33.236231 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.236116    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\") pod \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\" (UID: \"3a6a397c-526c-40be-9a53-1e1fd17bcff6\") "
Oct 30 07:55:33.237121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.237091    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca" pod "3a6a397c-526c-40be-9a53-1e1fd17bcff6" (UID: "3a6a397c-526c-40be-9a53-1e1fd17bcff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:55:33.240116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.240050    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz" pod "3a6a397c-526c-40be-9a53-1e1fd17bcff6" (UID: "3a6a397c-526c-40be-9a53-1e1fd17bcff6"). InnerVolumeSpecName "kube-api-access-28jdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:33.241031 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.240999    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca" pod "3a6a397c-526c-40be-9a53-1e1fd17bcff6" (UID: "3a6a397c-526c-40be-9a53-1e1fd17bcff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:55:33.338259 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.338225    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jdz\" (UniqueName: \"kubernetes.io/projected/3a6a397c-526c-40be-9a53-1e1fd17bcff6-kube-api-access-28jdz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:33.338462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.338260    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:33.338462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.338278    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a397c-526c-40be-9a53-1e1fd17bcff6-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:33.997074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.996983    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"
Oct 30 07:55:33.997074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.997025    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk" event={"ID":"3a6a397c-526c-40be-9a53-1e1fd17bcff6","Type":"ContainerDied","Data":"9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b"}
Oct 30 07:55:33.997649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.997076    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d858c6343719829f65bfce32ba6cd8b38ffd8b9b63fd76666b0a3e14ebfca0b"
Oct 30 07:55:33.998815 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.998790    6738 generic.go:356] "Generic (PLEG): container finished" podID="6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" containerID="6d637d36ff6dcd40536f634022213c2674aeb59ac5cdc2ebf18f2c9a1ea13fa3" exitCode=0
Oct 30 07:55:33.998964 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:33.998872    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4" event={"ID":"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30","Type":"ContainerDied","Data":"6d637d36ff6dcd40536f634022213c2674aeb59ac5cdc2ebf18f2c9a1ea13fa3"}
Oct 30 07:55:34.842309 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:34.842260    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:35.024803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.024757    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:35.211877 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.211851    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:35.280262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.280216    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\" (UniqueName: \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\") pod \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") "
Oct 30 07:55:35.280473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.280404    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\") pod \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") "
Oct 30 07:55:35.280473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.280457    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\") pod \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\" (UID: \"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30\") "
Oct 30 07:55:35.280962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.280929    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca" pod "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" (UID: "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:55:35.281122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.281083    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:35.282714 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.282691    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc" pod "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" (UID: "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30"). InnerVolumeSpecName "kube-api-access-94slc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:35.282854 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.282829    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca" pod "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" (UID: "6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:55:35.382209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.382164    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94slc\" (UniqueName: \"kubernetes.io/projected/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-kube-api-access-94slc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:35.382209 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:35.382204    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:36.017525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:36.017435    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4" event={"ID":"6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30","Type":"ContainerDied","Data":"9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30"}
Oct 30 07:55:36.017525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:36.017477    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"
Oct 30 07:55:36.017525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:36.017497    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b45ca6468d25aa6f8e2976b20fb02391b6c06a44329b3272f48750f9d4a4b30"
Oct 30 07:55:36.949155 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:36.949115    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:37.521152 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:55:37.521118    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:55:38.725885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:38.725841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:40.402863 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:40.402822    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:40.614772 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:40.614680    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:40.766803 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:40.766767    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:42.428334 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:42.428277    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:43.057419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.057379    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:43.585904 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.585866    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"]
Oct 30 07:55:43.586742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586714    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" containerName="revoke-keys"
Oct 30 07:55:43.586742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586742    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" containerName="revoke-keys"
Oct 30 07:55:43.586867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586792    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a397c-526c-40be-9a53-1e1fd17bcff6" containerName="revoke-keys"
Oct 30 07:55:43.586867 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586797    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a397c-526c-40be-9a53-1e1fd17bcff6" containerName="revoke-keys"
Oct 30 07:55:43.586937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586914    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a397c-526c-40be-9a53-1e1fd17bcff6" containerName="revoke-keys"
Oct 30 07:55:43.586937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.586927    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" containerName="revoke-keys"
Oct 30 07:55:43.591066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.591047    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.607008 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.606980    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"]
Oct 30 07:55:43.614601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.614570    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:55:43.614819 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.614632    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:55:43.720829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.720792    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9dn\" (UniqueName: \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.721021 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.720932    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.721137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.721121    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.822314 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.822269    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.822543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.822444    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9dn\" (UniqueName: \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.822543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.822533    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.823184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.823162    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.824735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.824712    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.846124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.846056    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9dn\" (UniqueName: \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\") pod \"maas-api-revoke-sub-ec983ff7-sbbjt\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:43.907379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:43.907340    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:44.239374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:44.239342    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"]
Oct 30 07:55:44.336627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:44.336592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:45.170087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:45.170046    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt" event={"ID":"cc97af42-5355-495c-b5bf-4bc52759ad46","Type":"ContainerStarted","Data":"4576154abe4590b2920fd1967d2e7f7a7e8ff97234139894549415df435ec712"}
Oct 30 07:55:45.170609 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:45.170117    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt" event={"ID":"cc97af42-5355-495c-b5bf-4bc52759ad46","Type":"ContainerStarted","Data":"c84630882f5cd1f7530613d1d3bac7103ec69cb30f44161f6686381d73ed3efa"}
Oct 30 07:55:45.214063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:45.214009    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt" podStartSLOduration=2.213992377 podStartE2EDuration="2.213992377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:55:45.210325757 +0000 UTC m=+3505.643089645" watchObservedRunningTime="2026-10-30 07:55:45.213992377 +0000 UTC m=+3505.646756269"
Oct 30 07:55:46.180019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:46.179980    6738 generic.go:356] "Generic (PLEG): container finished" podID="cc97af42-5355-495c-b5bf-4bc52759ad46" containerID="4576154abe4590b2920fd1967d2e7f7a7e8ff97234139894549415df435ec712" exitCode=0
Oct 30 07:55:46.180546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:46.180067    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt" event={"ID":"cc97af42-5355-495c-b5bf-4bc52759ad46","Type":"ContainerDied","Data":"4576154abe4590b2920fd1967d2e7f7a7e8ff97234139894549415df435ec712"}
Oct 30 07:55:46.200464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:46.200432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:46.972187 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:46.972145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:47.361575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.361548    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:47.484553 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.484513    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\") pod \"cc97af42-5355-495c-b5bf-4bc52759ad46\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") "
Oct 30 07:55:47.484850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.484831    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\") pod \"cc97af42-5355-495c-b5bf-4bc52759ad46\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") "
Oct 30 07:55:47.484932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.484889    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\" (UniqueName: \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\") pod \"cc97af42-5355-495c-b5bf-4bc52759ad46\" (UID: \"cc97af42-5355-495c-b5bf-4bc52759ad46\") "
Oct 30 07:55:47.485219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.485195    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca" pod "cc97af42-5355-495c-b5bf-4bc52759ad46" (UID: "cc97af42-5355-495c-b5bf-4bc52759ad46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:55:47.485509 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.485490    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:47.486742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.486716    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca" pod "cc97af42-5355-495c-b5bf-4bc52759ad46" (UID: "cc97af42-5355-495c-b5bf-4bc52759ad46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:55:47.486963 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.486938    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn" pod "cc97af42-5355-495c-b5bf-4bc52759ad46" (UID: "cc97af42-5355-495c-b5bf-4bc52759ad46"). InnerVolumeSpecName "kube-api-access-4s9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:47.587458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.587377    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9dn\" (UniqueName: \"kubernetes.io/projected/cc97af42-5355-495c-b5bf-4bc52759ad46-kube-api-access-4s9dn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:47.587458 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:47.587404    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc97af42-5355-495c-b5bf-4bc52759ad46-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:55:48.044502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:48.044456    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:48.198621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:48.198587    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"
Oct 30 07:55:48.198621 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:48.198606    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt" event={"ID":"cc97af42-5355-495c-b5bf-4bc52759ad46","Type":"ContainerDied","Data":"c84630882f5cd1f7530613d1d3bac7103ec69cb30f44161f6686381d73ed3efa"}
Oct 30 07:55:48.198851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:48.198656    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84630882f5cd1f7530613d1d3bac7103ec69cb30f44161f6686381d73ed3efa"
Oct 30 07:55:49.948152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:49.948111    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:51.944086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:51.944054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:53.754653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:53.754597    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:55.655949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:55.655908    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:57.412418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:57.412375    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:55:59.326182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:55:59.326092    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:01.047355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:01.047282    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:02.807925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:02.807889    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:04.756818 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:04.756741    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:05.033205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.033109    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:05.333852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.333770    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"]
Oct 30 07:56:05.334671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.334652    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cc97af42-5355-495c-b5bf-4bc52759ad46" containerName="revoke-keys"
Oct 30 07:56:05.334671 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.334671    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc97af42-5355-495c-b5bf-4bc52759ad46" containerName="revoke-keys"
Oct 30 07:56:05.334804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.334793    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc97af42-5355-495c-b5bf-4bc52759ad46" containerName="revoke-keys"
Oct 30 07:56:05.338887 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.338867    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.353790 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.353756    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:56:05.354601 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.354112    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:56:05.356664 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.356645    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"]
Oct 30 07:56:05.505824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.505788    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.506019 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.505854    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwpl\" (UniqueName: \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.506103 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.506080    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.607544 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.607505    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.607728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.607572    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwpl\" (UniqueName: \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.607778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.607727    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.608286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.608263    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.610282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.610260    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.631584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.631551    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwpl\" (UniqueName: \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\") pod \"maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.654373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.654336    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:05.978589 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:56:05.978557    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52740539_9c49_442d_863b_edd6885932be.slice/crio-b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1 WatchSource:0}: Error finding container b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1: Status 404 returned error can't find the container with id b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1
Oct 30 07:56:05.979462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:05.979438    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"]
Oct 30 07:56:06.392760 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:06.392717    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st" event={"ID":"52740539-9c49-442d-863b-edd6885932be","Type":"ContainerStarted","Data":"ab3c0cc1425afdbe86b85779f4fbd981bd54bab75a1008ddc85d4386ae45acaa"}
Oct 30 07:56:06.392946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:06.392794    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st" event={"ID":"52740539-9c49-442d-863b-edd6885932be","Type":"ContainerStarted","Data":"b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1"}
Oct 30 07:56:06.438454 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:06.438398    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st" podStartSLOduration=1.438381198 podStartE2EDuration="1.438381198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:06.43297316 +0000 UTC m=+3526.865737053" watchObservedRunningTime="2026-10-30 07:56:06.438381198 +0000 UTC m=+3526.871145090"
Oct 30 07:56:06.658524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:06.658441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:07.404217 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:07.404184    6738 generic.go:356] "Generic (PLEG): container finished" podID="52740539-9c49-442d-863b-edd6885932be" containerID="ab3c0cc1425afdbe86b85779f4fbd981bd54bab75a1008ddc85d4386ae45acaa" exitCode=0
Oct 30 07:56:07.404685 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:07.404272    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st" event={"ID":"52740539-9c49-442d-863b-edd6885932be","Type":"ContainerDied","Data":"ab3c0cc1425afdbe86b85779f4fbd981bd54bab75a1008ddc85d4386ae45acaa"}
Oct 30 07:56:07.521692 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:07.521654    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:56:08.577320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.577179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:08.634576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.634547    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:08.651816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.651776    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\" (UniqueName: \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\") pod \"52740539-9c49-442d-863b-edd6885932be\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") "
Oct 30 07:56:08.652014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.651847    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\") pod \"52740539-9c49-442d-863b-edd6885932be\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") "
Oct 30 07:56:08.652014 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.652006    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\") pod \"52740539-9c49-442d-863b-edd6885932be\" (UID: \"52740539-9c49-442d-863b-edd6885932be\") "
Oct 30 07:56:08.652459 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.652425    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca" pod "52740539-9c49-442d-863b-edd6885932be" (UID: "52740539-9c49-442d-863b-edd6885932be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:08.652728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.652702    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52740539-9c49-442d-863b-edd6885932be-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:08.654087 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.654062    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca" pod "52740539-9c49-442d-863b-edd6885932be" (UID: "52740539-9c49-442d-863b-edd6885932be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:08.654176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.654146    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl" pod "52740539-9c49-442d-863b-edd6885932be" (UID: "52740539-9c49-442d-863b-edd6885932be"). InnerVolumeSpecName "kube-api-access-wkwpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:08.754099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.754066    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwpl\" (UniqueName: \"kubernetes.io/projected/52740539-9c49-442d-863b-edd6885932be-kube-api-access-wkwpl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:08.754304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:08.754121    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52740539-9c49-442d-863b-edd6885932be-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:09.424726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:09.424699    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"
Oct 30 07:56:09.424934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:09.424728    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st" event={"ID":"52740539-9c49-442d-863b-edd6885932be","Type":"ContainerDied","Data":"b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1"}
Oct 30 07:56:09.424934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:09.424789    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a654be7731248a36d4e61126b4abd573ea06fa2880eb7b8cc8f3e660c704f1"
Oct 30 07:56:10.335678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:10.335643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:12.255496 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:12.255460    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:14.115915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:14.115880    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:15.891661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:15.891625    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:56:15.928724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:15.928689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:16.048473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:16.048439    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:17.837687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:17.837642    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:19.576507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:19.576473    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:20.892143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:20.892091    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing" containerID="cri-o://cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08" gracePeriod=25
Oct 30 07:56:21.558474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:21.558439    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:23.039985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.039936    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"]
Oct 30 07:56:23.054839 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.054805    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4f433b-rf6bk"]
Oct 30 07:56:23.095979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.095948    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:56:23.096219 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.096195    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" podUID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" containerName="maas-api" containerID="cri-o://8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4" gracePeriod=30
Oct 30 07:56:23.162366 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.162303    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:23.457206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.457175    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:56:23.544079 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.544035    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\") pod \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") "
Oct 30 07:56:23.544422 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.544382    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") pod \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") "
Oct 30 07:56:23.544578 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.544460    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\") pod \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") "
Oct 30 07:56:23.544692 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.544674    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\" (UniqueName: \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\") pod \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\" (UID: \"0c9ee0f5-d825-422f-b9ab-13404a17ce66\") "
Oct 30 07:56:23.569126 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.567860    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls" pod "0c9ee0f5-d825-422f-b9ab-13404a17ce66" (UID: "0c9ee0f5-d825-422f-b9ab-13404a17ce66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:23.593415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.571609    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca" pod "0c9ee0f5-d825-422f-b9ab-13404a17ce66" (UID: "0c9ee0f5-d825-422f-b9ab-13404a17ce66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:23.606561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.606511    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs" pod "0c9ee0f5-d825-422f-b9ab-13404a17ce66" (UID: "0c9ee0f5-d825-422f-b9ab-13404a17ce66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:23.619525 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.619466    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2" pod "0c9ee0f5-d825-422f-b9ab-13404a17ce66" (UID: "0c9ee0f5-d825-422f-b9ab-13404a17ce66"). InnerVolumeSpecName "kube-api-access-pd7g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:23.631220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.631179    6738 generic.go:356] "Generic (PLEG): container finished" podID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" containerID="8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4" exitCode=0
Oct 30 07:56:23.631464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.631246    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"
Oct 30 07:56:23.631464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.631267    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" event={"ID":"0c9ee0f5-d825-422f-b9ab-13404a17ce66","Type":"ContainerDied","Data":"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"}
Oct 30 07:56:23.631464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.631379    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54" event={"ID":"0c9ee0f5-d825-422f-b9ab-13404a17ce66","Type":"ContainerDied","Data":"4b152feb11c01758c10965c0306714879668e94fb751cf4e1f7f9ba6f65c6d90"}
Oct 30 07:56:23.631464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.631415    6738 scope.go:122] "RemoveContainer" containerID="8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"
Oct 30 07:56:23.648482 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.648393    6738 scope.go:122] "RemoveContainer" containerID="8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"
Oct 30 07:56:23.649159 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:23.649085    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4\": container with ID starting with 8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4 not found: ID does not exist" containerID="8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"
Oct 30 07:56:23.649159 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.649129    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4"} err="failed to get container status \"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4\": rpc error: code = NotFound desc = could not find container \"8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4\": container with ID starting with 8cfb2269a39fbd6740df97c644e21d421f83ed1e501c6e1bf2307ee980250bd4 not found: ID does not exist"
Oct 30 07:56:23.654266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.654238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:23.666893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.666866    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7g2\" (UniqueName: \"kubernetes.io/projected/0c9ee0f5-d825-422f-b9ab-13404a17ce66-kube-api-access-pd7g2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:23.666893 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.666893    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ee0f5-d825-422f-b9ab-13404a17ce66-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:23.667056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.666905    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:23.667056 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.666914    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9ee0f5-d825-422f-b9ab-13404a17ce66-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:23.694791 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.694761    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da28ff55-b341-4c3e-8cbc-4afce1dccf01" path="/var/lib/kubelet/pods/da28ff55-b341-4c3e-8cbc-4afce1dccf01/volumes"
Oct 30 07:56:23.695232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.695207    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:56:23.703521 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.703497    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c0e05-84598d854-ntj54"]
Oct 30 07:56:23.771278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.771240    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:23.917353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.897661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:23.931909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.931827    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"]
Oct 30 07:56:23.939604 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:23.939550    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c0e05-xszzk"]
Oct 30 07:56:24.419342 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:24.419288    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:24.553569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:24.553530    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:24.611778 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:24.611747    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"]
Oct 30 07:56:24.664573 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:24.664541    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e9723-dx2st"]
Oct 30 07:56:24.684622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:24.684549    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:25.229538 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.229508    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"]
Oct 30 07:56:25.230322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230284    6738 cpu_manager.go:414] "containerMap: removing container" podUID="52740539-9c49-442d-863b-edd6885932be" containerName="revoke-keys"
Oct 30 07:56:25.230322 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230312    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52740539-9c49-442d-863b-edd6885932be" containerName="revoke-keys"
Oct 30 07:56:25.230424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230347    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" containerName="maas-api"
Oct 30 07:56:25.230424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230352    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" containerName="maas-api"
Oct 30 07:56:25.230500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230481    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" containerName="maas-api"
Oct 30 07:56:25.230500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.230498    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="52740539-9c49-442d-863b-edd6885932be" containerName="revoke-keys"
Oct 30 07:56:25.235728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.235707    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.265247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.265220    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:56:25.265469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.265249    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:56:25.275970 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.275927    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"]
Oct 30 07:56:25.300789 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.300738    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.301005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.300916    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.301005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.300964    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nlr\" (UniqueName: \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.306874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.306835    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:25.402720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.402684    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.402907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.402731    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nlr\" (UniqueName: \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.402977 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.402965    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.403543 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.403523    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.405530 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.405508    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.427392 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.427366    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nlr\" (UniqueName: \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\") pod \"maas-api-revoke-sub-3f2cfe54-qhd5h\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.551032 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.550948    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:25.695781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.695752    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9ee0f5-d825-422f-b9ab-13404a17ce66" path="/var/lib/kubelet/pods/0c9ee0f5-d825-422f-b9ab-13404a17ce66/volumes"
Oct 30 07:56:25.696655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.696627    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6a397c-526c-40be-9a53-1e1fd17bcff6" path="/var/lib/kubelet/pods/3a6a397c-526c-40be-9a53-1e1fd17bcff6/volumes"
Oct 30 07:56:25.697189 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.697167    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52740539-9c49-442d-863b-edd6885932be" path="/var/lib/kubelet/pods/52740539-9c49-442d-863b-edd6885932be/volumes"
Oct 30 07:56:25.832449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.832326    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:25.860362 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:56:25.860329    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3e5173_b005_4a96_b7d3_354e24e4e945.slice/crio-c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386 WatchSource:0}: Error finding container c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386: Status 404 returned error can't find the container with id c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386
Oct 30 07:56:25.863413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:25.863382    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"]
Oct 30 07:56:26.685271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:26.685214    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h" event={"ID":"da3e5173-b005-4a96-b7d3-354e24e4e945","Type":"ContainerStarted","Data":"fb91bb3d94bff17061a4a7d3c52f91866e79644599fab0110dd89c1c22648e56"}
Oct 30 07:56:26.685770 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:26.685285    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h" event={"ID":"da3e5173-b005-4a96-b7d3-354e24e4e945","Type":"ContainerStarted","Data":"c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386"}
Oct 30 07:56:26.729357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:26.729289    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h" podStartSLOduration=1.7292752660000001 podStartE2EDuration="1.729275266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:26.727482489 +0000 UTC m=+3547.160246382" watchObservedRunningTime="2026-10-30 07:56:26.729275266 +0000 UTC m=+3547.162039159"
Oct 30 07:56:26.733013 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:26.732977    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:27.702428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:27.702391    6738 generic.go:356] "Generic (PLEG): container finished" podID="da3e5173-b005-4a96-b7d3-354e24e4e945" containerID="fb91bb3d94bff17061a4a7d3c52f91866e79644599fab0110dd89c1c22648e56" exitCode=0
Oct 30 07:56:27.702898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:27.702465    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h" event={"ID":"da3e5173-b005-4a96-b7d3-354e24e4e945","Type":"ContainerDied","Data":"fb91bb3d94bff17061a4a7d3c52f91866e79644599fab0110dd89c1c22648e56"}
Oct 30 07:56:28.626111 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.626077    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:28.882623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.882558    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:28.960986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.960935    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\" (UniqueName: \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\") pod \"da3e5173-b005-4a96-b7d3-354e24e4e945\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") "
Oct 30 07:56:28.961176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.961045    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\") pod \"da3e5173-b005-4a96-b7d3-354e24e4e945\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") "
Oct 30 07:56:28.961176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.961159    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\") pod \"da3e5173-b005-4a96-b7d3-354e24e4e945\" (UID: \"da3e5173-b005-4a96-b7d3-354e24e4e945\") "
Oct 30 07:56:28.961631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.961599    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca" pod "da3e5173-b005-4a96-b7d3-354e24e4e945" (UID: "da3e5173-b005-4a96-b7d3-354e24e4e945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:28.962143 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.962117    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:28.963381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.963348    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr" pod "da3e5173-b005-4a96-b7d3-354e24e4e945" (UID: "da3e5173-b005-4a96-b7d3-354e24e4e945"). InnerVolumeSpecName "kube-api-access-l7nlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:28.963746 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:28.963728    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca" pod "da3e5173-b005-4a96-b7d3-354e24e4e945" (UID: "da3e5173-b005-4a96-b7d3-354e24e4e945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:29.064190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.064149    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nlr\" (UniqueName: \"kubernetes.io/projected/da3e5173-b005-4a96-b7d3-354e24e4e945-kube-api-access-l7nlr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:29.064190 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.064184    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3e5173-b005-4a96-b7d3-354e24e4e945-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:29.725523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.725493    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"
Oct 30 07:56:29.726237 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.725526    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h" event={"ID":"da3e5173-b005-4a96-b7d3-354e24e4e945","Type":"ContainerDied","Data":"c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386"}
Oct 30 07:56:29.726388 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.726271    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c53e12aa2d78b1f9edef13815de021e512c0a0de0bc154bff85ef3b0e9ea8386"
Oct 30 07:56:29.805982 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.805947    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:29.940872 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:29.940826    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:30.061884 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:30.061793    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:30.578072 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:30.578032    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:30.690939 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:30.690902    6738 scope.go:122] "RemoveContainer" containerID="02a11d057db9fd55a76eed7ad3e2d01c07610f8075243606869b0bc0155e8958"
Oct 30 07:56:30.702497 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:30.702475    6738 scope.go:122] "RemoveContainer" containerID="fd571c331833444869ed84059c3abf5d089584c18f4711bb10b13453a0099d51"
Oct 30 07:56:30.713405 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:30.713385    6738 scope.go:122] "RemoveContainer" containerID="69bdd75c0a9ba1a4404a847cc7d5b295e040d4438eb1cb853ecbee026df1b926"
Oct 30 07:56:31.225206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:31.225165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:31.362409 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:31.362374    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:32.592724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:32.592682    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:32.736321 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:32.733940    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:32.851116 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:32.851025    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:32.962396 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:32.962362    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:56:32.962632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:32.962575    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy" containerID="cri-o://6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d" gracePeriod=30
Oct 30 07:56:33.014341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.014279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:33.249146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.249109    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:56:33.249656 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.249626    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy" containerID="cri-o://4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820" gracePeriod=30
Oct 30 07:56:33.279345 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.279316    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:33.584886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.584801    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"]
Oct 30 07:56:33.585661 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.585646    6738 cpu_manager.go:414] "containerMap: removing container" podUID="da3e5173-b005-4a96-b7d3-354e24e4e945" containerName="revoke-keys"
Oct 30 07:56:33.585724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.585664    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e5173-b005-4a96-b7d3-354e24e4e945" containerName="revoke-keys"
Oct 30 07:56:33.585784 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.585773    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e5173-b005-4a96-b7d3-354e24e4e945" containerName="revoke-keys"
Oct 30 07:56:33.591267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.591249    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.606655 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.606623    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:56:33.607040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.606895    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:56:33.607908 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.607888    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"]
Oct 30 07:56:33.684036 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.684000    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.684224 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.684065    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.684399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.684279    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9hg\" (UniqueName: \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.785541 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.785505    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.785786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.785733    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9hg\" (UniqueName: \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.786214 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.786189    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.786832 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.786810    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.788005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.787985    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.809750 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.809715    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9hg\" (UniqueName: \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\") pod \"maas-api-revoke-sub-7cdefef5-t94sd\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.846068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.845977    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"]
Oct 30 07:56:33.851075 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.851050    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.869607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.869575    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"]
Oct 30 07:56:33.887461 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.887420    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqfz\" (UniqueName: \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.887702 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.887682    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.887799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.887777    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.906767 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.906733    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:33.992674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.991044    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.992674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.991132    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.992674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.991354    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqfz\" (UniqueName: \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.993232 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.993150    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:33.993512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:33.993480    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:34.000850 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.000825    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:34.016569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.016541    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqfz\" (UniqueName: \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\") pod \"maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:34.117608 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.117521    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:34.168650 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.168618    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:34.219508 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:56:34.219470    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895de860_6819_4d00_9d65_bf321d2685cf.slice/crio-36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc WatchSource:0}: Error finding container 36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc: Status 404 returned error can't find the container with id 36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc
Oct 30 07:56:34.219943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.219918    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"]
Oct 30 07:56:34.492804 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.489074    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"]
Oct 30 07:56:34.784252 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.784213    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd" event={"ID":"895de860-6819-4d00-9d65-bf321d2685cf","Type":"ContainerStarted","Data":"b4c265cf171698e44d3147755fbb1b2a8cb2e3ee53fa81b47869703579dcff2e"}
Oct 30 07:56:34.784728 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.784278    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd" event={"ID":"895de860-6819-4d00-9d65-bf321d2685cf","Type":"ContainerStarted","Data":"36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc"}
Oct 30 07:56:34.785775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.785741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d" event={"ID":"1d1e0cd9-4d8e-4254-a9be-a965524bb99b","Type":"ContainerStarted","Data":"2e8188a17d488dcb32fea0416d3c8b330c2d663cf36724cec4e2bb1e53b17dd4"}
Oct 30 07:56:34.841749 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.841689    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd" podStartSLOduration=1.841673093 podStartE2EDuration="1.841673093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:34.838700898 +0000 UTC m=+3555.271464793" watchObservedRunningTime="2026-10-30 07:56:34.841673093 +0000 UTC m=+3555.274437304"
Oct 30 07:56:34.911313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:34.910661    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:35.055607 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.055513    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:35.796081 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.795994    6738 generic.go:356] "Generic (PLEG): container finished" podID="895de860-6819-4d00-9d65-bf321d2685cf" containerID="b4c265cf171698e44d3147755fbb1b2a8cb2e3ee53fa81b47869703579dcff2e" exitCode=0
Oct 30 07:56:35.796523 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.796072    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd" event={"ID":"895de860-6819-4d00-9d65-bf321d2685cf","Type":"ContainerDied","Data":"b4c265cf171698e44d3147755fbb1b2a8cb2e3ee53fa81b47869703579dcff2e"}
Oct 30 07:56:35.797922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.797900    6738 generic.go:356] "Generic (PLEG): container finished" podID="1d1e0cd9-4d8e-4254-a9be-a965524bb99b" containerID="d813037fe2eb0e07b0ea22d3890d7510112c539ab59fcab6e2a4d65ab4a70852" exitCode=0
Oct 30 07:56:35.798058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.797941    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d" event={"ID":"1d1e0cd9-4d8e-4254-a9be-a965524bb99b","Type":"ContainerDied","Data":"d813037fe2eb0e07b0ea22d3890d7510112c539ab59fcab6e2a4d65ab4a70852"}
Oct 30 07:56:35.818472 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:35.818431    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:36.260733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.260690    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:36.854080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.854044    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:36.854564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.854239    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"]
Oct 30 07:56:36.861063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.860166    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:36.874431 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.874315    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"]
Oct 30 07:56:36.959277 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.959240    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:36.959526 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.959505    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:36.959613 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:36.959599    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwpt\" (UniqueName: \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.061962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.061279    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwpt\" (UniqueName: \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.061962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.061486    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.061962 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.061736    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.064105 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.062653    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.069419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.069396    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.086040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.086013    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwpt\" (UniqueName: \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\") pod \"maas-api-revoke-sub-ed4f193e-nq287\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.114504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.114476    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:37.117335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.117311    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:37.163440 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163407    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\") pod \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") "
Oct 30 07:56:37.163624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163452    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\") pod \"895de860-6819-4d00-9d65-bf321d2685cf\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") "
Oct 30 07:56:37.163624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163472    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\" (UniqueName: \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\") pod \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") "
Oct 30 07:56:37.163624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163548    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\" (UniqueName: \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\") pod \"895de860-6819-4d00-9d65-bf321d2685cf\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") "
Oct 30 07:56:37.163883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163795    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\") pod \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\" (UID: \"1d1e0cd9-4d8e-4254-a9be-a965524bb99b\") "
Oct 30 07:56:37.163883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163841    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\") pod \"895de860-6819-4d00-9d65-bf321d2685cf\" (UID: \"895de860-6819-4d00-9d65-bf321d2685cf\") "
Oct 30 07:56:37.163883 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.163865    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca" pod "1d1e0cd9-4d8e-4254-a9be-a965524bb99b" (UID: "1d1e0cd9-4d8e-4254-a9be-a965524bb99b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:37.164355 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.164283    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca" pod "895de860-6819-4d00-9d65-bf321d2685cf" (UID: "895de860-6819-4d00-9d65-bf321d2685cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:37.164572 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.164553    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.164644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.164575    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.166401 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.166369    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz" pod "1d1e0cd9-4d8e-4254-a9be-a965524bb99b" (UID: "1d1e0cd9-4d8e-4254-a9be-a965524bb99b"). InnerVolumeSpecName "kube-api-access-nhqfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:37.166513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.166458    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca" pod "895de860-6819-4d00-9d65-bf321d2685cf" (UID: "895de860-6819-4d00-9d65-bf321d2685cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:37.166794 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.166768    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg" pod "895de860-6819-4d00-9d65-bf321d2685cf" (UID: "895de860-6819-4d00-9d65-bf321d2685cf"). InnerVolumeSpecName "kube-api-access-ms9hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:37.167701 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.167673    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca" pod "1d1e0cd9-4d8e-4254-a9be-a965524bb99b" (UID: "1d1e0cd9-4d8e-4254-a9be-a965524bb99b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:37.190473 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.190442    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:37.266419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.266381    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895de860-6819-4d00-9d65-bf321d2685cf-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.266419 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.266418    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqfz\" (UniqueName: \"kubernetes.io/projected/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-kube-api-access-nhqfz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.266713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.266435    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9hg\" (UniqueName: \"kubernetes.io/projected/895de860-6819-4d00-9d65-bf321d2685cf-kube-api-access-ms9hg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.266713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.266451    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1e0cd9-4d8e-4254-a9be-a965524bb99b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:37.520685 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:37.520646    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:56:37.532313 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:56:37.532257    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62943fed_8d63_4332_9309_5e89eb45fd2b.slice/crio-3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e WatchSource:0}: Error finding container 3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e: Status 404 returned error can't find the container with id 3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e
Oct 30 07:56:37.533437 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.533414    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"]
Oct 30 07:56:37.819393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.819307    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"
Oct 30 07:56:37.819393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.819321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd" event={"ID":"895de860-6819-4d00-9d65-bf321d2685cf","Type":"ContainerDied","Data":"36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc"}
Oct 30 07:56:37.819393 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.819377    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36646facc1be83aeb60ab79c53242853c74dfd9e3981ceb760cf251e4942bcbc"
Oct 30 07:56:37.821185 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.821163    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"
Oct 30 07:56:37.821364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.821186    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d" event={"ID":"1d1e0cd9-4d8e-4254-a9be-a965524bb99b","Type":"ContainerDied","Data":"2e8188a17d488dcb32fea0416d3c8b330c2d663cf36724cec4e2bb1e53b17dd4"}
Oct 30 07:56:37.821364 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.821231    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8188a17d488dcb32fea0416d3c8b330c2d663cf36724cec4e2bb1e53b17dd4"
Oct 30 07:56:37.822885 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:37.822857    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287" event={"ID":"62943fed-8d63-4332-9309-5e89eb45fd2b","Type":"ContainerStarted","Data":"3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e"}
Oct 30 07:56:38.058101 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:38.058066    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023aac0b_5913_4781_b327_909efbb5fe24.slice/crio-conmon-6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023aac0b_5913_4781_b327_909efbb5fe24.slice/crio-6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 07:56:38.257618 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.257591    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:56:38.281726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281692    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.281910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281787    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.281910 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281822    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281918    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281958    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.281963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.282062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282002    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282070    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.282286 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282261    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282287    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.282418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282345    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:38.282524 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282460    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282536    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\") pod \"023aac0b-5913-4781-b327-909efbb5fe24\" (UID: \"023aac0b-5913-4781-b327-909efbb5fe24\") "
Oct 30 07:56:38.282584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.282460    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.283451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.283425    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.283451 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.283452    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.283672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.283467    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023aac0b-5913-4781-b327-909efbb5fe24-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.283672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.283481    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.283672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.283494    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.284403 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.284376    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:38.284480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.284419    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "kube-api-access-gnfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:38.284480 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.284461    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.284669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.284654    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo" pod "023aac0b-5913-4781-b327-909efbb5fe24" (UID: "023aac0b-5913-4781-b327-909efbb5fe24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:56:38.384725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.384685    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfgs\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-kube-api-access-gnfgs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.384725 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.384725    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023aac0b-5913-4781-b327-909efbb5fe24-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.384965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.384738    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023aac0b-5913-4781-b327-909efbb5fe24-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.384965 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.384750    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023aac0b-5913-4781-b327-909efbb5fe24-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.541756 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.541725    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:56:38.588005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.587961    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588006    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588077    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588151    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588271    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588342    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.588500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588445    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.588500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588478    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.589110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588629    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.589110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588654    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.589110 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.588993    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.590443    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.590671    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.591481    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\") pod \"10021836-b642-4e73-9574-83d9b4f654a3\" (UID: \"10021836-b642-4e73-9574-83d9b4f654a3\") "
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.592395    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10021836-b642-4e73-9574-83d9b4f654a3-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.592418    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.592433    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.592448    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.592647 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.592462    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.593705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.593004    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:56:38.593705 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.593257    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:38.594755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.594728    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "kube-api-access-7hwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:38.594909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.594889    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy" pod "10021836-b642-4e73-9574-83d9b4f654a3" (UID: "10021836-b642-4e73-9574-83d9b4f654a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:38.694215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.694178    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwrb\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-kube-api-access-7hwrb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.694215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.694214    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10021836-b642-4e73-9574-83d9b4f654a3-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.694215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.694224    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10021836-b642-4e73-9574-83d9b4f654a3-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.694581 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.694235    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10021836-b642-4e73-9574-83d9b4f654a3-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:38.740659 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.740615    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:38.832843 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.832811    6738 generic.go:356] "Generic (PLEG): container finished" podID="023aac0b-5913-4781-b327-909efbb5fe24" containerID="6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d" exitCode=0
Oct 30 07:56:38.833068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.832861    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"
Oct 30 07:56:38.833068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.832896    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerDied","Data":"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"}
Oct 30 07:56:38.833068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.832987    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996" event={"ID":"023aac0b-5913-4781-b327-909efbb5fe24","Type":"ContainerDied","Data":"6fc5b2a31fe9b5f13ec15cc8d8557c590a362efbd5aac1b0c68a0cbcf0c4a3d7"}
Oct 30 07:56:38.833068 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.833017    6738 scope.go:122] "RemoveContainer" containerID="6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"
Oct 30 07:56:38.834959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.834930    6738 generic.go:356] "Generic (PLEG): container finished" podID="62943fed-8d63-4332-9309-5e89eb45fd2b" containerID="88b88956743cd6a3aa8632ce75b86c288abec9f7e671c75be5960361fd5dfb20" exitCode=0
Oct 30 07:56:38.834959 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.834953    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287" event={"ID":"62943fed-8d63-4332-9309-5e89eb45fd2b","Type":"ContainerDied","Data":"88b88956743cd6a3aa8632ce75b86c288abec9f7e671c75be5960361fd5dfb20"}
Oct 30 07:56:38.837368 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.837346    6738 generic.go:356] "Generic (PLEG): container finished" podID="10021836-b642-4e73-9574-83d9b4f654a3" containerID="4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820" exitCode=0
Oct 30 07:56:38.837512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.837412    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"
Oct 30 07:56:38.837512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.837443    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerDied","Data":"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"}
Oct 30 07:56:38.837512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.837501    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66" event={"ID":"10021836-b642-4e73-9574-83d9b4f654a3","Type":"ContainerDied","Data":"b109089334c79460470818c5bb8b682b7432a902b3638ec21551fec3a92eb662"}
Oct 30 07:56:38.848768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.848741    6738 scope.go:122] "RemoveContainer" containerID="1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"
Oct 30 07:56:38.859592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.859567    6738 scope.go:122] "RemoveContainer" containerID="6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"
Oct 30 07:56:38.859899 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:38.859879    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d\": container with ID starting with 6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d not found: ID does not exist" containerID="6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"
Oct 30 07:56:38.859948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.859909    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d"} err="failed to get container status \"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d\": rpc error: code = NotFound desc = could not find container \"6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d\": container with ID starting with 6f395db6cc17243e0e394e89012c4232fb0b69f63ec1a7428eda5742076e762d not found: ID does not exist"
Oct 30 07:56:38.859948 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.859926    6738 scope.go:122] "RemoveContainer" containerID="1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"
Oct 30 07:56:38.860183 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:38.860162    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2\": container with ID starting with 1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2 not found: ID does not exist" containerID="1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"
Oct 30 07:56:38.860220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.860189    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2"} err="failed to get container status \"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2\": rpc error: code = NotFound desc = could not find container \"1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2\": container with ID starting with 1c2136d66f853fa574a078e499e9f1cabd76bf16085bb491356ebd3d22b5a0a2 not found: ID does not exist"
Oct 30 07:56:38.860220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.860207    6738 scope.go:122] "RemoveContainer" containerID="4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"
Oct 30 07:56:38.871485 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.871465    6738 scope.go:122] "RemoveContainer" containerID="a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"
Oct 30 07:56:38.882192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.882007    6738 scope.go:122] "RemoveContainer" containerID="4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"
Oct 30 07:56:38.882375 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:38.882349    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820\": container with ID starting with 4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820 not found: ID does not exist" containerID="4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"
Oct 30 07:56:38.882445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.882385    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820"} err="failed to get container status \"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820\": rpc error: code = NotFound desc = could not find container \"4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820\": container with ID starting with 4bf53ffd73d983b02429532dec8c4438e104a9537940bca85e0a722c3802c820 not found: ID does not exist"
Oct 30 07:56:38.882445 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.882404    6738 scope.go:122] "RemoveContainer" containerID="a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"
Oct 30 07:56:38.882692 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:38.882677    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443\": container with ID starting with a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443 not found: ID does not exist" containerID="a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"
Oct 30 07:56:38.882743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.882694    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443"} err="failed to get container status \"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443\": rpc error: code = NotFound desc = could not find container \"a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443\": container with ID starting with a304199a726a3e4780974fa1f91ae5efc135939bdf5765ec4ef0c099fc429443 not found: ID does not exist"
Oct 30 07:56:38.931104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.931069    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:56:38.934873 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.934849    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0e9723-openshift-default-65b7bddc6c-2lc66"]
Oct 30 07:56:38.968051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.968016    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:56:38.978331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:38.978286    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8c0e05-openshift-default-568cf5f-gd996"]
Oct 30 07:56:39.692308 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:39.692266    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023aac0b-5913-4781-b327-909efbb5fe24" path="/var/lib/kubelet/pods/023aac0b-5913-4781-b327-909efbb5fe24/volumes"
Oct 30 07:56:39.692898 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:39.692882    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10021836-b642-4e73-9574-83d9b4f654a3" path="/var/lib/kubelet/pods/10021836-b642-4e73-9574-83d9b4f654a3/volumes"
Oct 30 07:56:40.014201 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.014174    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:40.115874 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.115840    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\") pod \"62943fed-8d63-4332-9309-5e89eb45fd2b\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") "
Oct 30 07:56:40.116071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.115902    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\") pod \"62943fed-8d63-4332-9309-5e89eb45fd2b\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") "
Oct 30 07:56:40.116071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.116010    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\" (UniqueName: \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\") pod \"62943fed-8d63-4332-9309-5e89eb45fd2b\" (UID: \"62943fed-8d63-4332-9309-5e89eb45fd2b\") "
Oct 30 07:56:40.116352 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.116281    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca" pod "62943fed-8d63-4332-9309-5e89eb45fd2b" (UID: "62943fed-8d63-4332-9309-5e89eb45fd2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:40.116515 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.116498    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:40.118005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.117985    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca" pod "62943fed-8d63-4332-9309-5e89eb45fd2b" (UID: "62943fed-8d63-4332-9309-5e89eb45fd2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:40.118121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.118108    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt" pod "62943fed-8d63-4332-9309-5e89eb45fd2b" (UID: "62943fed-8d63-4332-9309-5e89eb45fd2b"). InnerVolumeSpecName "kube-api-access-shwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:40.218207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.218175    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62943fed-8d63-4332-9309-5e89eb45fd2b-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:40.218207 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.218206    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwpt\" (UniqueName: \"kubernetes.io/projected/62943fed-8d63-4332-9309-5e89eb45fd2b-kube-api-access-shwpt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:40.646335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.646283    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:40.859167 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.859135    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"
Oct 30 07:56:40.859722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.859175    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287" event={"ID":"62943fed-8d63-4332-9309-5e89eb45fd2b","Type":"ContainerDied","Data":"3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e"}
Oct 30 07:56:40.859722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:40.859231    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3491b979292bf9a4ea85a70c1f88a54c308e2a5a00978b6556e5a7b7ac60695e"
Oct 30 07:56:41.164986 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.164920    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:56:41.235313 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.235266    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\" (UniqueName: \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\") pod \"cca08662-4a18-48b5-9ab3-a203f79f8f29\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") "
Oct 30 07:56:41.235490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.235379    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\") pod \"cca08662-4a18-48b5-9ab3-a203f79f8f29\" (UID: \"cca08662-4a18-48b5-9ab3-a203f79f8f29\") "
Oct 30 07:56:41.235808 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.235776    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume" pod "cca08662-4a18-48b5-9ab3-a203f79f8f29" (UID: "cca08662-4a18-48b5-9ab3-a203f79f8f29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:41.236192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.236157    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca08662-4a18-48b5-9ab3-a203f79f8f29-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:41.237829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.237806    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw" pod "cca08662-4a18-48b5-9ab3-a203f79f8f29" (UID: "cca08662-4a18-48b5-9ab3-a203f79f8f29"). InnerVolumeSpecName "kube-api-access-sxdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:41.337895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.337852    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdgw\" (UniqueName: \"kubernetes.io/projected/cca08662-4a18-48b5-9ab3-a203f79f8f29-kube-api-access-sxdgw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:41.869104 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.869068    6738 generic.go:356] "Generic (PLEG): container finished" podID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerID="cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08" exitCode=0
Oct 30 07:56:41.869528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.869123    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"
Oct 30 07:56:41.869528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.869150    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" event={"ID":"cca08662-4a18-48b5-9ab3-a203f79f8f29","Type":"ContainerDied","Data":"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"}
Oct 30 07:56:41.869528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.869211    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd" event={"ID":"cca08662-4a18-48b5-9ab3-a203f79f8f29","Type":"ContainerDied","Data":"de84ac9da89c14bbc3df35a7a44ad8fcea89550c7b505ddbd8752957f15ae337"}
Oct 30 07:56:41.869528 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.869237    6738 scope.go:122] "RemoveContainer" containerID="cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"
Oct 30 07:56:41.880713 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.880686    6738 scope.go:122] "RemoveContainer" containerID="cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"
Oct 30 07:56:41.880998 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:41.880980    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08\": container with ID starting with cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08 not found: ID does not exist" containerID="cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"
Oct 30 07:56:41.881051 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.881006    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08"} err="failed to get container status \"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08\": rpc error: code = NotFound desc = could not find container \"cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08\": container with ID starting with cb0f8972667a028a43b9260905a2781ae5f2a1e1e0219230632ab65f6be95f08 not found: ID does not exist"
Oct 30 07:56:41.930545 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.930510    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:56:41.942460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:41.942426    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e9723-5b554c7759-qrdgd"]
Oct 30 07:56:42.464918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:42.464877    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:43.692305 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:43.692262    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" path="/var/lib/kubelet/pods/cca08662-4a18-48b5-9ab3-a203f79f8f29/volumes"
Oct 30 07:56:44.004273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.004170    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:44.227993 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.227956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:44.304846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.304744    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:56:44.457582 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.457537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:44.587027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.586929    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:44.877516 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:44.877486    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:46.034571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:46.034536    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:46.250842 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:46.250803    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:48.031988 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:48.031951    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"]
Oct 30 07:56:48.043922 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:48.043891    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0617966-vpc7k"]
Oct 30 07:56:48.127048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:48.127012    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:49.305576 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.305526    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" podUID="75bd152f-53b5-436b-b75e-954f4a3c6429" containerName="payload-processing" containerID="cri-o://af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3" gracePeriod=25
Oct 30 07:56:49.611551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.611519    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:56:49.695162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.695126    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de995859-a91b-41e3-8eb5-32fb607639d5" path="/var/lib/kubelet/pods/de995859-a91b-41e3-8eb5-32fb607639d5/volumes"
Oct 30 07:56:49.776704 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.776666    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\") pod \"75bd152f-53b5-436b-b75e-954f4a3c6429\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") "
Oct 30 07:56:49.776938 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.776868    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\" (UniqueName: \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\") pod \"75bd152f-53b5-436b-b75e-954f4a3c6429\" (UID: \"75bd152f-53b5-436b-b75e-954f4a3c6429\") "
Oct 30 07:56:49.777085 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.777061    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume" pod "75bd152f-53b5-436b-b75e-954f4a3c6429" (UID: "75bd152f-53b5-436b-b75e-954f4a3c6429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:49.777534 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.777517    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75bd152f-53b5-436b-b75e-954f4a3c6429-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:49.779017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.778992    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966" pod "75bd152f-53b5-436b-b75e-954f4a3c6429" (UID: "75bd152f-53b5-436b-b75e-954f4a3c6429"). InnerVolumeSpecName "kube-api-access-gx966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:49.879029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.878993    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx966\" (UniqueName: \"kubernetes.io/projected/75bd152f-53b5-436b-b75e-954f4a3c6429-kube-api-access-gx966\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:49.940923 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.940888    6738 generic.go:356] "Generic (PLEG): container finished" podID="75bd152f-53b5-436b-b75e-954f4a3c6429" containerID="af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3" exitCode=0
Oct 30 07:56:49.941112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.940936    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"
Oct 30 07:56:49.941112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.940962    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" event={"ID":"75bd152f-53b5-436b-b75e-954f4a3c6429","Type":"ContainerDied","Data":"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"}
Oct 30 07:56:49.941112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.941027    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c" event={"ID":"75bd152f-53b5-436b-b75e-954f4a3c6429","Type":"ContainerDied","Data":"fef4c2bdd60f5a86c3abeea5e21f5058bbefe52e3f8b7e1dd150188d987dc733"}
Oct 30 07:56:49.941112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.941056    6738 scope.go:122] "RemoveContainer" containerID="af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"
Oct 30 07:56:49.953402 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.953383    6738 scope.go:122] "RemoveContainer" containerID="af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"
Oct 30 07:56:49.953725 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:56:49.953701    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3\": container with ID starting with af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3 not found: ID does not exist" containerID="af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"
Oct 30 07:56:49.953780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:49.953739    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3"} err="failed to get container status \"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3\": rpc error: code = NotFound desc = could not find container \"af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3\": container with ID starting with af1b7d30eab5b880a1501f84e24375603aedb5803b70c0b34aedddd96aa4ede3 not found: ID does not exist"
Oct 30 07:56:50.039351 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:50.039314    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:56:50.043757 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:50.043725    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8514a3-89fd46658-r569c"]
Oct 30 07:56:50.096562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:50.096527    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:50.265743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:50.265638    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:51.693853 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:51.693816    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bd152f-53b5-436b-b75e-954f4a3c6429" path="/var/lib/kubelet/pods/75bd152f-53b5-436b-b75e-954f4a3c6429/volumes"
Oct 30 07:56:51.878145 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:51.878100    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:53.269940 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:53.269903    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:53.882093 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:53.882055    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:55.025319 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.024839    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.341751    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"]
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.342985    6738 cpu_manager.go:414] "containerMap: removing container" podUID="895de860-6819-4d00-9d65-bf321d2685cf" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343009    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895de860-6819-4d00-9d65-bf321d2685cf" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343044    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343053    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343074    6738 cpu_manager.go:414] "containerMap: removing container" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343082    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343126    6738 cpu_manager.go:414] "containerMap: removing container" podUID="75bd152f-53b5-436b-b75e-954f4a3c6429" containerName="payload-processing"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343134    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bd152f-53b5-436b-b75e-954f4a3c6429" containerName="payload-processing"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343149    6738 cpu_manager.go:414] "containerMap: removing container" podUID="62943fed-8d63-4332-9309-5e89eb45fd2b" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343157    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62943fed-8d63-4332-9309-5e89eb45fd2b" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343171    6738 cpu_manager.go:414] "containerMap: removing container" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343179    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343203    6738 cpu_manager.go:414] "containerMap: removing container" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343210    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343223    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e0cd9-4d8e-4254-a9be-a965524bb99b" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343230    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e0cd9-4d8e-4254-a9be-a965524bb99b" containerName="revoke-keys"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343258    6738 cpu_manager.go:414] "containerMap: removing container" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.343370 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.343264    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344326    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344352    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344367    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e0cd9-4d8e-4254-a9be-a965524bb99b" containerName="revoke-keys"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344382    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="023aac0b-5913-4781-b327-909efbb5fe24" containerName="istio-proxy"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344395    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="895de860-6819-4d00-9d65-bf321d2685cf" containerName="revoke-keys"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344411    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca08662-4a18-48b5-9ab3-a203f79f8f29" containerName="payload-pre-processing"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344425    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="10021836-b642-4e73-9574-83d9b4f654a3" containerName="istio-proxy"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344441    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bd152f-53b5-436b-b75e-954f4a3c6429" containerName="payload-processing"
Oct 30 07:56:55.344639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.344452    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="62943fed-8d63-4332-9309-5e89eb45fd2b" containerName="revoke-keys"
Oct 30 07:56:55.352681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.352654    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.367678 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.367646    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"]
Oct 30 07:56:55.372633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.372601    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:56:55.373615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.373592    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:56:55.493331 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.477428    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnk4g\" (UniqueName: \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.493780 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.493742    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.536864 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.536808    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.643064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.642070    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.643064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.642263    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnk4g\" (UniqueName: \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.643064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.642719    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.643064 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.642973    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.646920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.646899    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.666184 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.666158    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnk4g\" (UniqueName: \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\") pod \"maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:55.691316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:55.690516    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:56.202723 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:56:56.202644    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695c09af_a037_47ca_b9ab_5b9ea034ac11.slice/crio-cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6 WatchSource:0}: Error finding container cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6: Status 404 returned error can't find the container with id cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6
Oct 30 07:56:56.203121 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:56.202725    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"]
Oct 30 07:56:56.258065 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:56.258028    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2" event={"ID":"695c09af-a037-47ca-b9ab-5b9ea034ac11","Type":"ContainerStarted","Data":"cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6"}
Oct 30 07:56:56.352899 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:56.352861    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:57.267729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:57.267695    6738 generic.go:356] "Generic (PLEG): container finished" podID="695c09af-a037-47ca-b9ab-5b9ea034ac11" containerID="8eed75af3e73bf30671702dc6004e2bea4e45b96a33ac541ac7edf53aa87aed7" exitCode=0
Oct 30 07:56:57.268182 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:57.267784    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2" event={"ID":"695c09af-a037-47ca-b9ab-5b9ea034ac11","Type":"ContainerDied","Data":"8eed75af3e73bf30671702dc6004e2bea4e45b96a33ac541ac7edf53aa87aed7"}
Oct 30 07:56:58.353629 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.353321    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:56:58.470675 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.470644    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:58.505592 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.505560    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\" (UniqueName: \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\") pod \"695c09af-a037-47ca-b9ab-5b9ea034ac11\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") "
Oct 30 07:56:58.505812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.505606    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\") pod \"695c09af-a037-47ca-b9ab-5b9ea034ac11\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") "
Oct 30 07:56:58.505812 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.505768    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\") pod \"695c09af-a037-47ca-b9ab-5b9ea034ac11\" (UID: \"695c09af-a037-47ca-b9ab-5b9ea034ac11\") "
Oct 30 07:56:58.506074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.506052    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca" pod "695c09af-a037-47ca-b9ab-5b9ea034ac11" (UID: "695c09af-a037-47ca-b9ab-5b9ea034ac11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:58.506424 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.506406    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:58.507769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.507747    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca" pod "695c09af-a037-47ca-b9ab-5b9ea034ac11" (UID: "695c09af-a037-47ca-b9ab-5b9ea034ac11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:58.507937 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.507917    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g" pod "695c09af-a037-47ca-b9ab-5b9ea034ac11" (UID: "695c09af-a037-47ca-b9ab-5b9ea034ac11"). InnerVolumeSpecName "kube-api-access-wnk4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:58.608323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.608224    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c09af-a037-47ca-b9ab-5b9ea034ac11-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:58.608323 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:58.608257    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnk4g\" (UniqueName: \"kubernetes.io/projected/695c09af-a037-47ca-b9ab-5b9ea034ac11-kube-api-access-wnk4g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:56:59.285806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:59.285770    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"
Oct 30 07:56:59.285806 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:59.285798    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2" event={"ID":"695c09af-a037-47ca-b9ab-5b9ea034ac11","Type":"ContainerDied","Data":"cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6"}
Oct 30 07:56:59.286039 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:56:59.285844    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc1b6da3e1bb4f5dbb03b85eeb8f84232d579e139e9615d40ffa24e47b79ccf6"
Oct 30 07:57:00.183071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:00.183027    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:02.020623 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:02.020577    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:03.807985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:03.807945    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:05.691795 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:05.691755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:05.899759 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:05.899718    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:57:06.020344 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:06.020214    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:06.140208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:06.140168    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:07.522973 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:07.522937    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:57:07.535564 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:07.535525    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:09.328580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:09.328538    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:11.006016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:11.005970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:11.166125 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:11.166091    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:11.346546 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:11.346424    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing" containerID="cri-o://746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b" gracePeriod=25
Oct 30 07:57:11.800413 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:11.800315    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:11.928215 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:11.928176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:13.240644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:13.240595    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:13.935264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:13.935221    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:57:15.164606 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:15.164564    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:15.629720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:15.629686    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:15.767160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:15.767120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:17.206271 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:17.206237    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:17.485668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:17.485580    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:17.619233 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:17.619197    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:19.125829 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:19.125792    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:20.967398 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:20.967361    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:22.865598 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:22.865561    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:23.935686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:23.935648    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:57:24.641054 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:24.641019    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:26.284258 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:26.284217    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:26.941220 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:26.941176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:27.062506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.062465    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:27.114684 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.114649    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"]
Oct 30 07:57:27.118594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.118563    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8514a3-c7g6d"]
Oct 30 07:57:27.192584 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.192497    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:27.612372 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.612261    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:27.692432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:27.692397    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1e0cd9-4d8e-4254-a9be-a965524bb99b" path="/var/lib/kubelet/pods/1d1e0cd9-4d8e-4254-a9be-a965524bb99b/volumes"
Oct 30 07:57:29.218732 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:29.218644    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:30.637755 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.637715    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:30.758627 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.758588    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:30.776285 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.776244    6738 scope.go:122] "RemoveContainer" containerID="42d2322054fc66880bfe637774fb597cd09b938cba1ec1b9238436887f1876b9"
Oct 30 07:57:30.790262 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.790234    6738 scope.go:122] "RemoveContainer" containerID="048e2ecc34e6ff7b63a8befb6c05e586810b0324aac85cb49afb12a6207e8876"
Oct 30 07:57:30.866115 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.866079    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:57:30.866327 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.866286    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" podUID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" containerName="maas-api" containerID="cri-o://1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd" gracePeriod=30
Oct 30 07:57:30.958428 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:30.958330    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:31.866002 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.864118    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:31.898833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.898796    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:57:31.958953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.957866    6738 generic.go:356] "Generic (PLEG): container finished" podID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerID="746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b" exitCode=0
Oct 30 07:57:31.958953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.958098    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" event={"ID":"ec2d94de-6a65-4372-b772-bd65f09cae53","Type":"ContainerDied","Data":"746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b"}
Oct 30 07:57:31.962660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.962027    6738 generic.go:356] "Generic (PLEG): container finished" podID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" containerID="1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd" exitCode=0
Oct 30 07:57:31.962660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.962101    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"
Oct 30 07:57:31.962660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.962084    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" event={"ID":"e756f6be-9b0f-4a2b-b361-3ee56382a7cf","Type":"ContainerDied","Data":"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"}
Oct 30 07:57:31.962660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.962184    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s" event={"ID":"e756f6be-9b0f-4a2b-b361-3ee56382a7cf","Type":"ContainerDied","Data":"ca75cb01f1aba505ca53b7137c06aa5806e9376a15cbcc565e10ad4ec32f089c"}
Oct 30 07:57:31.962660 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:31.962215    6738 scope.go:122] "RemoveContainer" containerID="1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"
Oct 30 07:57:32.062320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.061921    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") pod \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") "
Oct 30 07:57:32.062320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.061988    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\") pod \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") "
Oct 30 07:57:32.062320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.062071    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\" (UniqueName: \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\") pod \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") "
Oct 30 07:57:32.062320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.062168    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\") pod \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\" (UID: \"e756f6be-9b0f-4a2b-b361-3ee56382a7cf\") "
Oct 30 07:57:32.239320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.221956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:32.239320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.222721    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca" pod "e756f6be-9b0f-4a2b-b361-3ee56382a7cf" (UID: "e756f6be-9b0f-4a2b-b361-3ee56382a7cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:32.355341 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.347116    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.432900 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.427955    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx" pod "e756f6be-9b0f-4a2b-b361-3ee56382a7cf" (UID: "e756f6be-9b0f-4a2b-b361-3ee56382a7cf"). InnerVolumeSpecName "kube-api-access-q2sjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.447433    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.455916    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls" pod "e756f6be-9b0f-4a2b-b361-3ee56382a7cf" (UID: "e756f6be-9b0f-4a2b-b361-3ee56382a7cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.456129    6738 scope.go:122] "RemoveContainer" containerID="1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.463444    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"]
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.482194    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.482369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.482230    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2sjx\" (UniqueName: \"kubernetes.io/projected/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-kube-api-access-q2sjx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.490161 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:32.489885    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd\": container with ID starting with 1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd not found: ID does not exist" containerID="1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"
Oct 30 07:57:32.490161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.489953    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd"} err="failed to get container status \"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd\": rpc error: code = NotFound desc = could not find container \"1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd\": container with ID starting with 1ef2ebb8c60456c593f5a5d1bea7e45eda0b8eb3fe60e4cd7d4be3b2c8ea4abd not found: ID does not exist"
Oct 30 07:57:32.560833 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.559562    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs" pod "e756f6be-9b0f-4a2b-b361-3ee56382a7cf" (UID: "e756f6be-9b0f-4a2b-b361-3ee56382a7cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:57:32.605802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.604946    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e756f6be-9b0f-4a2b-b361-3ee56382a7cf-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.607635 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.607536    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccac6f-w9pt2"]
Oct 30 07:57:32.617627 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:32.617585    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d94de_6a65_4372_b772_bd65f09cae53.slice/crio-conmon-746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d94de_6a65_4372_b772_bd65f09cae53.slice/crio-746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 07:57:32.632212 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.632116    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:57:32.657436 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.655036    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:32.661710 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.661645    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:57:32.665658 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.665631    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ccac6f-56c5566d5d-k6z9s"]
Oct 30 07:57:32.714353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.714322    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\") pod \"ec2d94de-6a65-4372-b772-bd65f09cae53\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") "
Oct 30 07:57:32.714632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.714601    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\" (UniqueName: \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\") pod \"ec2d94de-6a65-4372-b772-bd65f09cae53\" (UID: \"ec2d94de-6a65-4372-b772-bd65f09cae53\") "
Oct 30 07:57:32.731192 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.715740    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume" pod "ec2d94de-6a65-4372-b772-bd65f09cae53" (UID: "ec2d94de-6a65-4372-b772-bd65f09cae53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:32.750418 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.739004    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d94de-6a65-4372-b772-bd65f09cae53-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.752369 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.750820    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg" pod "ec2d94de-6a65-4372-b772-bd65f09cae53" (UID: "ec2d94de-6a65-4372-b772-bd65f09cae53"). InnerVolumeSpecName "kube-api-access-99kqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:32.855005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:32.854780    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99kqg\" (UniqueName: \"kubernetes.io/projected/ec2d94de-6a65-4372-b772-bd65f09cae53-kube-api-access-99kqg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:33.039603 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.039476    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx" event={"ID":"ec2d94de-6a65-4372-b772-bd65f09cae53","Type":"ContainerDied","Data":"a5eb78b27ac58064499fac64dc7b6f390e995c3f672738bae93a9b3915180b3c"}
Oct 30 07:57:33.040247 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.040223    6738 scope.go:122] "RemoveContainer" containerID="746f6375c0c786651e18e0ecffec6ebbbb2ed0b3465b6d114f8bd72ae939b29b"
Oct 30 07:57:33.040610 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.040591    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"
Oct 30 07:57:33.123320 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.120210    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:57:33.149654 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.149609    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ccac6f-66d7b59bf9-2wglx"]
Oct 30 07:57:33.692905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.692825    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695c09af-a037-47ca-b9ab-5b9ea034ac11" path="/var/lib/kubelet/pods/695c09af-a037-47ca-b9ab-5b9ea034ac11/volumes"
Oct 30 07:57:33.693361 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.693346    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" path="/var/lib/kubelet/pods/e756f6be-9b0f-4a2b-b361-3ee56382a7cf/volumes"
Oct 30 07:57:33.693724 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:33.693711    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" path="/var/lib/kubelet/pods/ec2d94de-6a65-4372-b772-bd65f09cae53/volumes"
Oct 30 07:57:34.374038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:34.373992    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:34.576741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:34.576707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:34.693197 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:34.693109    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:34.831768 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:34.819625    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:35.150512 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:35.150480    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:35.774775 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:35.774741    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:35.909427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:35.909390    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:36.147161 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:36.147126    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:36.406208 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:36.406106    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:37.520377 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:37.520320    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:57:38.251646 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:38.251612    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:39.309506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:39.309477    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:39.500254 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:39.500219    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:39.702268 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:39.702235    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:57:39.702508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:39.702485    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy" containerID="cri-o://6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a" gracePeriod=30
Oct 30 07:57:39.735510 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:39.735471    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:40.280336 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.280289    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"]
Oct 30 07:57:40.281060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281044    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing"
Oct 30 07:57:40.281113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281061    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing"
Oct 30 07:57:40.281113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281081    6738 cpu_manager.go:414] "containerMap: removing container" podUID="695c09af-a037-47ca-b9ab-5b9ea034ac11" containerName="revoke-keys"
Oct 30 07:57:40.281113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281088    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695c09af-a037-47ca-b9ab-5b9ea034ac11" containerName="revoke-keys"
Oct 30 07:57:40.281218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281127    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" containerName="maas-api"
Oct 30 07:57:40.281218 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281133    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" containerName="maas-api"
Oct 30 07:57:40.281283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281239    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d94de-6a65-4372-b772-bd65f09cae53" containerName="payload-pre-processing"
Oct 30 07:57:40.281283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281250    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="695c09af-a037-47ca-b9ab-5b9ea034ac11" containerName="revoke-keys"
Oct 30 07:57:40.281283 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.281266    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e756f6be-9b0f-4a2b-b361-3ee56382a7cf" containerName="maas-api"
Oct 30 07:57:40.286667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.286647    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.303348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.303282    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"]
Oct 30 07:57:40.314486 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.314450    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:57:40.314946 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.314525    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:57:40.323206 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.323174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:40.399415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.399373    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.399830 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.399788    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.400280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.400255    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbd7\" (UniqueName: \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.502722 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.502676    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbd7\" (UniqueName: \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.502975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.502791    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.502975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.502876    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.503622 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.503595    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.505282 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.505256    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.526911 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.526877    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbd7\" (UniqueName: \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\") pod \"maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.607343 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.607214    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:40.918470 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:40.918439    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"]
Oct 30 07:57:40.918470 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:57:40.918462    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9597963_e226_4f03_9579_dfc08ebf615f.slice/crio-c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7 WatchSource:0}: Error finding container c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7: Status 404 returned error can't find the container with id c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7
Oct 30 07:57:41.119412 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:41.119378    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7" event={"ID":"a9597963-e226-4f03-9579-dfc08ebf615f","Type":"ContainerStarted","Data":"c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7"}
Oct 30 07:57:42.129246 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:42.129211    6738 generic.go:356] "Generic (PLEG): container finished" podID="a9597963-e226-4f03-9579-dfc08ebf615f" containerID="25f7dc47b4effb39b81756742dcee2a1b16e89f4652d55d212b225c9b0157eca" exitCode=0
Oct 30 07:57:42.129667 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:42.129311    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7" event={"ID":"a9597963-e226-4f03-9579-dfc08ebf615f","Type":"ContainerDied","Data":"25f7dc47b4effb39b81756742dcee2a1b16e89f4652d55d212b225c9b0157eca"}
Oct 30 07:57:42.255052 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:42.254989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:42.511513 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:42.511418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:42.635569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:42.635524    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:43.304741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.304711    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:43.455809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.455717    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\") pod \"a9597963-e226-4f03-9579-dfc08ebf615f\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") "
Oct 30 07:57:43.455999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.455917    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\" (UniqueName: \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\") pod \"a9597963-e226-4f03-9579-dfc08ebf615f\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") "
Oct 30 07:57:43.455999 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.455983    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\") pod \"a9597963-e226-4f03-9579-dfc08ebf615f\" (UID: \"a9597963-e226-4f03-9579-dfc08ebf615f\") "
Oct 30 07:57:43.456156 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.456111    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca" pod "a9597963-e226-4f03-9579-dfc08ebf615f" (UID: "a9597963-e226-4f03-9579-dfc08ebf615f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:43.456490 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.456473    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:43.458016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.457991    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7" pod "a9597963-e226-4f03-9579-dfc08ebf615f" (UID: "a9597963-e226-4f03-9579-dfc08ebf615f"). InnerVolumeSpecName "kube-api-access-4gbd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:43.458176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.458156    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca" pod "a9597963-e226-4f03-9579-dfc08ebf615f" (UID: "a9597963-e226-4f03-9579-dfc08ebf615f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:57:43.557918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.557873    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbd7\" (UniqueName: \"kubernetes.io/projected/a9597963-e226-4f03-9579-dfc08ebf615f-kube-api-access-4gbd7\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:43.557918 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:43.557908    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9597963-e226-4f03-9579-dfc08ebf615f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:44.147168 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.147139    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"
Oct 30 07:57:44.147395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.147165    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7" event={"ID":"a9597963-e226-4f03-9579-dfc08ebf615f","Type":"ContainerDied","Data":"c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7"}
Oct 30 07:57:44.147395 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.147214    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a17b61f3417e6dae0499a49bf3cb2d3fb57ed49d4a7bc60e1b2566734568c7"
Oct 30 07:57:44.249023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.248983    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:44.373797 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.373761    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:44.997339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:44.997307    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:57:45.083109 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083064    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083135    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083269    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083353 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083326    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083417    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:45.083559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083448    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083483    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083559 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083525    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083592    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.083787 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.083683    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\") pod \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\" (UID: \"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1\") "
Oct 30 07:57:45.084235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084073    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:45.084235 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084219    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084501    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084950    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084972    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.084988    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.085001    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.085074 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.085013    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.086657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.086614    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:45.086824 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.086802    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:45.086944 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.086926    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:57:45.087140 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.087111    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89" pod "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" (UID: "d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1"). InnerVolumeSpecName "kube-api-access-bzz89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:45.169047 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.169014    6738 generic.go:356] "Generic (PLEG): container finished" podID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerID="6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a" exitCode=0
Oct 30 07:57:45.169267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.169086    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"
Oct 30 07:57:45.169267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.169094    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerDied","Data":"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"}
Oct 30 07:57:45.169267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.169186    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9" event={"ID":"d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1","Type":"ContainerDied","Data":"d5a76b0ecba1de444753368083be083e7ab29ddc9853b3dfef6f80492610978d"}
Oct 30 07:57:45.169267 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.169213    6738 scope.go:122] "RemoveContainer" containerID="6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"
Oct 30 07:57:45.185968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.185937    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzz89\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-kube-api-access-bzz89\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.185968 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.185966    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.186172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.185982    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.186172 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.185995    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:57:45.188612 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.187254    6738 scope.go:122] "RemoveContainer" containerID="4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"
Oct 30 07:57:45.202255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.202231    6738 scope.go:122] "RemoveContainer" containerID="6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"
Oct 30 07:57:45.202642 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:45.202608    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a\": container with ID starting with 6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a not found: ID does not exist" containerID="6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"
Oct 30 07:57:45.202715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.202657    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a"} err="failed to get container status \"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a\": rpc error: code = NotFound desc = could not find container \"6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a\": container with ID starting with 6c03a7316bdb365dca40e79968b87785bc5a1cd36bc5ab8c59d449c08962f87a not found: ID does not exist"
Oct 30 07:57:45.202715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.202683    6738 scope.go:122] "RemoveContainer" containerID="4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"
Oct 30 07:57:45.202983 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:57:45.202961    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d\": container with ID starting with 4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d not found: ID does not exist" containerID="4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"
Oct 30 07:57:45.203063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.202989    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d"} err="failed to get container status \"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d\": rpc error: code = NotFound desc = could not find container \"4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d\": container with ID starting with 4bafd981f25bdc6b9b6b8bf8bcd0156a6ec7a5c92b8fc09d3a26386fccf3585d not found: ID does not exist"
Oct 30 07:57:45.283681 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.283595    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:57:45.308160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.308122    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ccac6f-openshift-default-758c47dd99-n4jn9"]
Oct 30 07:57:45.692478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:45.692443    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" path="/var/lib/kubelet/pods/d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1/volumes"
Oct 30 07:57:46.192223 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:46.192183    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:48.054076 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:48.054041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:49.948275 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:49.948234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:51.138113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:51.138062    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:51.261695 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:51.261498    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:51.843758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:51.843718    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:53.708998 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:53.708961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:55.460851 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:55.460755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:57.534846 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:57.534807    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:57:59.419549 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:57:59.419487    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:01.463625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:01.463581    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:03.391735 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:03.391698    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:03.781849 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:03.781766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:05.201765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:05.201733    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:07.112852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:07.112813    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:07.521029 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:58:07.520987    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:58:08.786720 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.786683    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 07:58:08.787743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787718    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a9597963-e226-4f03-9579-dfc08ebf615f" containerName="revoke-keys"
Oct 30 07:58:08.787743 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787739    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9597963-e226-4f03-9579-dfc08ebf615f" containerName="revoke-keys"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787774    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787779    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787786    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787791    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787925    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787933    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b0a9e9-6fcd-476a-9436-61efa2e9d1b1" containerName="istio-proxy"
Oct 30 07:58:08.787975 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.787948    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9597963-e226-4f03-9579-dfc08ebf615f" containerName="revoke-keys"
Oct 30 07:58:08.791715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.791696    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.806876 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.806851    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-083c13ed-serving-cert\""
Oct 30 07:58:08.829335 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.826602    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 07:58:08.866718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.866686    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6zj\" (UniqueName: \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.866943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.866759    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.866943 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.866883    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.867055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.866994    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.882594 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.882554    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:08.968248 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.968209    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6zj\" (UniqueName: \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.968476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.968325    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.968476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.968368    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.968476 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.968418    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.969146 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.969119    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.970969 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.970939    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.971077 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.970977    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.992687 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.992652    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6zj\" (UniqueName: \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\") pod \"maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:08.999561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:08.999528    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:09.106555 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:09.106451    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:09.114005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:09.113976    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:09.415765 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:09.415736    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 07:58:09.417082 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:58:09.417055    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9393ee_d855_4c96_83de_b2d039f93356.slice/crio-97df71fc46fb7340007b75b51da06efc369be3b4527405d9b28b9abce80bfb37 WatchSource:0}: Error finding container 97df71fc46fb7340007b75b51da06efc369be3b4527405d9b28b9abce80bfb37: Status 404 returned error can't find the container with id 97df71fc46fb7340007b75b51da06efc369be3b4527405d9b28b9abce80bfb37
Oct 30 07:58:09.418697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:09.418675    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:58:10.423029 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:10.422985    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" event={"ID":"8a9393ee-d855-4c96-83de-b2d039f93356","Type":"ContainerStarted","Data":"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"}
Oct 30 07:58:10.423571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:10.423058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" event={"ID":"8a9393ee-d855-4c96-83de-b2d039f93356","Type":"ContainerStarted","Data":"97df71fc46fb7340007b75b51da06efc369be3b4527405d9b28b9abce80bfb37"}
Oct 30 07:58:10.423571 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:10.423089    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:10.936934 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:10.936895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:11.389464 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:11.389425    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:12.844825 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:12.844787    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:14.618312 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.618231    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" podStartSLOduration=6.209668503 podStartE2EDuration="6.618211491s" totalImagesPullingTime="408.542988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:10.488528707 +0000 UTC m=+3650.921292639" watchObservedRunningTime="2026-10-30 07:58:14.618211491 +0000 UTC m=+3655.050975383"
Oct 30 07:58:14.622099 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.622072    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"]
Oct 30 07:58:14.627174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.627153    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.658434 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.658398    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"]
Oct 30 07:58:14.665500 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.665466    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:58:14.665719 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.665619    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:58:14.674932 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.674899    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.675174 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.675040    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nh78\" (UniqueName: \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.675288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.675268    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.689575 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.689317    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:14.776415 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.776372    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nh78\" (UniqueName: \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.776633 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.776478    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.776729 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.776706    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.777835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.777807    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.782347 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.779759    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.801160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.801127    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nh78\" (UniqueName: \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\") pod \"maas-api-revoke-sub-2ff40750-whtpm\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:14.946200 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:14.946166    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:15.256686 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:15.256654    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"]
Oct 30 07:58:15.256898 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:58:15.256875    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6585be74_ee6f_4a06_ba42_e31ec3e660ae.slice/crio-378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e WatchSource:0}: Error finding container 378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e: Status 404 returned error can't find the container with id 378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e
Oct 30 07:58:15.465796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:15.465760    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm" event={"ID":"6585be74-ee6f-4a06-ba42-e31ec3e660ae","Type":"ContainerStarted","Data":"378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e"}
Oct 30 07:58:16.438848 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:16.438800    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 07:58:16.475921 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:16.475885    6738 generic.go:356] "Generic (PLEG): container finished" podID="6585be74-ee6f-4a06-ba42-e31ec3e660ae" containerID="19e4f994130dfc2a8350c3ae12674631f6ec1f9022719c54b7365456445f9eae" exitCode=0
Oct 30 07:58:16.476119 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:16.475977    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm" event={"ID":"6585be74-ee6f-4a06-ba42-e31ec3e660ae","Type":"ContainerDied","Data":"19e4f994130dfc2a8350c3ae12674631f6ec1f9022719c54b7365456445f9eae"}
Oct 30 07:58:16.521276 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:16.521236    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:17.641040 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.641016    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:17.773980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.773886    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\") pod \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") "
Oct 30 07:58:17.774160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.773990    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\" (UniqueName: \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\") pod \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") "
Oct 30 07:58:17.774160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.774042    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\") pod \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\" (UID: \"6585be74-ee6f-4a06-ba42-e31ec3e660ae\") "
Oct 30 07:58:17.774348 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.774290    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca" pod "6585be74-ee6f-4a06-ba42-e31ec3e660ae" (UID: "6585be74-ee6f-4a06-ba42-e31ec3e660ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:58:17.774682 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.774658    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:17.776221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.776191    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca" pod "6585be74-ee6f-4a06-ba42-e31ec3e660ae" (UID: "6585be74-ee6f-4a06-ba42-e31ec3e660ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:58:17.776382 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.776263    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78" pod "6585be74-ee6f-4a06-ba42-e31ec3e660ae" (UID: "6585be74-ee6f-4a06-ba42-e31ec3e660ae"). InnerVolumeSpecName "kube-api-access-2nh78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:17.876399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.876365    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nh78\" (UniqueName: \"kubernetes.io/projected/6585be74-ee6f-4a06-ba42-e31ec3e660ae-kube-api-access-2nh78\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:17.876399 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:17.876395    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585be74-ee6f-4a06-ba42-e31ec3e660ae-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:18.356672 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:18.356634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:18.494752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:18.494720    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm" event={"ID":"6585be74-ee6f-4a06-ba42-e31ec3e660ae","Type":"ContainerDied","Data":"378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e"}
Oct 30 07:58:18.494949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:18.494765    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"
Oct 30 07:58:18.494949 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:18.494770    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378918d400bbd8b432140e2d372fdd91e5e0e3a230f55149cb73b7e9a900635e"
Oct 30 07:58:20.149520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:20.149484    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:22.028017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:22.027970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:23.788080 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:23.788040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:25.068625 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:25.068589    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:25.648796 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:25.648757    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:27.467715 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:27.467676    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:29.227698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:29.227651    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:30.879628 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:30.879594    6738 scope.go:122] "RemoveContainer" containerID="eec3d0ff11821d6bce19443886beb5d1c2896ecc6bf472fa0dc8d00f94e60b0c"
Oct 30 07:58:30.891520 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:30.891491    6738 scope.go:122] "RemoveContainer" containerID="af1d2399eeefd61642b6cacfa4f06ea289d908a5f2ca6e36af4c695bd65a411a"
Oct 30 07:58:31.057417 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:31.057384    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:31.565117 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:31.565077    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:32.925669 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:32.925630    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:34.679251 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:34.679213    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:35.321432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.321388    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:35.447199 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.447158    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:35.534632 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.534595    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"]
Oct 30 07:58:35.545430 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.545396    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-896ca4-9wzs7"]
Oct 30 07:58:35.717443 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.717406    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9597963-e226-4f03-9579-dfc08ebf615f" path="/var/lib/kubelet/pods/a9597963-e226-4f03-9579-dfc08ebf615f/volumes"
Oct 30 07:58:35.734950 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.734841    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:35.888202 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:35.888161    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:36.102406 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:36.101859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:37.520497 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:58:37.520459    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:58:37.801560 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:37.801462    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:38.310909 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:38.310868    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:39.636045 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:39.636011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:41.450191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:41.450153    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:42.749273 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:42.749238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:42.876905 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:42.876858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:43.162152 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:43.162118    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:43.779995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:43.779952    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:44.126551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:44.126520    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:44.445777 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:44.445676    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:44.665665 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:44.665625    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:46.128120 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:46.128088    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:47.030861 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:47.030825    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"]
Oct 30 07:58:47.042782 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:47.042750    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3a32a9-kmc6g"]
Oct 30 07:58:47.692009 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:47.691972    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec" path="/var/lib/kubelet/pods/cf5e265c-2d27-42f6-9fa4-5f9fa11b8aec/volumes"
Oct 30 07:58:47.945631 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:47.945542    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:49.847821 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:49.847787    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:50.991670 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:50.991634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:51.123593 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.123560    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:51.316086 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.315999    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"]
Oct 30 07:58:51.316915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.316889    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6585be74-ee6f-4a06-ba42-e31ec3e660ae" containerName="revoke-keys"
Oct 30 07:58:51.316915 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.316912    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6585be74-ee6f-4a06-ba42-e31ec3e660ae" containerName="revoke-keys"
Oct 30 07:58:51.317162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.317147    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6585be74-ee6f-4a06-ba42-e31ec3e660ae" containerName="revoke-keys"
Oct 30 07:58:51.322238 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.322214    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.341071 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.341039    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"]
Oct 30 07:58:51.353340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.348895    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:58:51.353340 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.348961    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:58:51.484003 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.483962    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.484203 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.484118    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgn5\" (UniqueName: \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.484266 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.484202    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.585326 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.585207    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.585504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.585425    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.585586 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.585570    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgn5\" (UniqueName: \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.586053 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.586026    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.587845 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.587821    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.610569 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.610539    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgn5\" (UniqueName: \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.637339 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.637272    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:51.888316 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.886692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:51.946023 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:51.945990    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"]
Oct 30 07:58:51.946239 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:58:51.946219    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1b89b7_fc3f_48aa_ae36_5317b7ab2a42.slice/crio-7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1 WatchSource:0}: Error finding container 7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1: Status 404 returned error can't find the container with id 7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1
Oct 30 07:58:52.939653 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:52.939615    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz" event={"ID":"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42","Type":"ContainerStarted","Data":"ae52ab4243fb99def4495dc66b5c893414aa098a8a246bce8f14ba6cc677bef4"}
Oct 30 07:58:52.940073 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:52.939699    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz" event={"ID":"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42","Type":"ContainerStarted","Data":"7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1"}
Oct 30 07:58:52.981698 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:52.981643    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz" podStartSLOduration=1.9816291289999999 podStartE2EDuration="1.981629129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:52.980074688 +0000 UTC m=+3693.412838579" watchObservedRunningTime="2026-10-30 07:58:52.981629129 +0000 UTC m=+3693.414393371"
Oct 30 07:58:53.767280 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:53.767238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:53.948799 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:53.948759    6738 generic.go:356] "Generic (PLEG): container finished" podID="4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" containerID="ae52ab4243fb99def4495dc66b5c893414aa098a8a246bce8f14ba6cc677bef4" exitCode=0
Oct 30 07:58:53.949205 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:53.948847    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz" event={"ID":"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42","Type":"ContainerDied","Data":"ae52ab4243fb99def4495dc66b5c893414aa098a8a246bce8f14ba6cc677bef4"}
Oct 30 07:58:54.858239 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:54.858198    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:55.227644 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.227615    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:55.241668 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.241630    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\") pod \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") "
Oct 30 07:58:55.241852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.241782    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\" (UniqueName: \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\") pod \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") "
Oct 30 07:58:55.241852 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.241837    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\") pod \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\" (UID: \"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42\") "
Oct 30 07:58:55.242027 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.242002    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca" pod "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" (UID: "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:58:55.242381 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.242361    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:55.244532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.244504    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca" pod "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" (UID: "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:58:55.244532 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.244524    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5" pod "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" (UID: "4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42"). InnerVolumeSpecName "kube-api-access-lsgn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:55.343835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.343801    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsgn5\" (UniqueName: \"kubernetes.io/projected/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-kube-api-access-lsgn5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:55.343835 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.343870    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:58:55.737191 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.737157    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:55.966022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.965943    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"
Oct 30 07:58:55.966022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.965980    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz" event={"ID":"4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42","Type":"ContainerDied","Data":"7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1"}
Oct 30 07:58:55.966221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:55.966027    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f557faef486dea22348b1e9ade5e48f2d1392804791e034f83da453925e0db1"
Oct 30 07:58:57.522985 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:57.522950    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:58:59.339741 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:58:59.339660    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:01.368317 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:01.355872    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:01.806504 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:01.806469    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:59:01.852274 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:01.852242    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:59:01.960637 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:01.960604    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:02.031634 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:02.031603    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"]
Oct 30 07:59:02.035462 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:02.035430    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1f5bb69-kk7jv"]
Oct 30 07:59:02.095089 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:02.094977    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:03.391751 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:03.391710    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:03.692048 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:03.691952    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004e9b0-6808-41d1-a132-783dd254a6f5" path="/var/lib/kubelet/pods/f004e9b0-6808-41d1-a132-783dd254a6f5/volumes"
Oct 30 07:59:05.261742 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:05.261706    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:06.807493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:06.807457    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing" containerID="cri-o://c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749" gracePeriod=25
Oct 30 07:59:06.853615 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:06.853570    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing" containerID="cri-o://db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c" gracePeriod=25
Oct 30 07:59:07.021141 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:07.021110    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:07.520692 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:07.520648    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:59:08.513063 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:08.513026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:08.900933 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:08.900895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:09.485663 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:09.485619    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:09.689580 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:09.689547    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:10.109122 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:10.109084    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:10.859591 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:10.859549    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:12.686953 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:12.686921    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:14.030506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:14.030466    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"]
Oct 30 07:59:14.042718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:14.042685    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bf9076-zdngt"]
Oct 30 07:59:14.557379 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:14.557340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:15.693176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:15.693136    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e29a031-3c6f-4952-a7e2-32bd8090a0ec" path="/var/lib/kubelet/pods/1e29a031-3c6f-4952-a7e2-32bd8090a0ec/volumes"
Oct 30 07:59:15.717798 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:15.717761    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:59:15.718004 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:15.717817    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:59:16.438060 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:16.438028    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:17.758502 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:17.758464    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:18.454865 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:18.454820    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:20.353176 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:20.353135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:20.638365 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:20.638330    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:22.325058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:22.325020    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:24.141802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:24.141766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:25.718391 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:25.718356    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:59:25.718894 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:25.718368    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:59:25.960160 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:25.960122    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:27.146397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.146368    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:59:27.173137 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.173110    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:59:27.179995 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.179964    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\") pod \"4b48ee79-9976-42f2-8c53-5a13f51a056f\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") "
Oct 30 07:59:27.180173 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.180128    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\" (UniqueName: \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\") pod \"4b48ee79-9976-42f2-8c53-5a13f51a056f\" (UID: \"4b48ee79-9976-42f2-8c53-5a13f51a056f\") "
Oct 30 07:59:27.180374 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.180351    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume" pod "4b48ee79-9976-42f2-8c53-5a13f51a056f" (UID: "4b48ee79-9976-42f2-8c53-5a13f51a056f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:59:27.180763 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.180743    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b48ee79-9976-42f2-8c53-5a13f51a056f-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:27.182162 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.182140    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f" pod "4b48ee79-9976-42f2-8c53-5a13f51a056f" (UID: "4b48ee79-9976-42f2-8c53-5a13f51a056f"). InnerVolumeSpecName "kube-api-access-cd25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:27.269427 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.269387    6738 generic.go:356] "Generic (PLEG): container finished" podID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerID="c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749" exitCode=0
Oct 30 07:59:27.269639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.269454    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"
Oct 30 07:59:27.269639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.269466    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" event={"ID":"4b48ee79-9976-42f2-8c53-5a13f51a056f","Type":"ContainerDied","Data":"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"}
Oct 30 07:59:27.269639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.269534    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k" event={"ID":"4b48ee79-9976-42f2-8c53-5a13f51a056f","Type":"ContainerDied","Data":"90fe6f16e29173f850784dad7a4e49d8d25a5a01879cb336803639fe86827be7"}
Oct 30 07:59:27.269639 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.269582    6738 scope.go:122] "RemoveContainer" containerID="c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"
Oct 30 07:59:27.271244 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.271221    6738 generic.go:356] "Generic (PLEG): container finished" podID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerID="db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c" exitCode=0
Oct 30 07:59:27.271478 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.271460    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" event={"ID":"f7efff85-fd9e-4e94-a124-8f535c889d80","Type":"ContainerDied","Data":"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"}
Oct 30 07:59:27.271561 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.271508    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94" event={"ID":"f7efff85-fd9e-4e94-a124-8f535c889d80","Type":"ContainerDied","Data":"574a44b74b95e62f8a72d060c6f34dd28e689fa748f6fcc71cb2fbce29f5b874"}
Oct 30 07:59:27.271624 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.271586    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"
Oct 30 07:59:27.281585 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.281542    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\" (UniqueName: \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\") pod \"f7efff85-fd9e-4e94-a124-8f535c889d80\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") "
Oct 30 07:59:27.281731 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.281630    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\") pod \"f7efff85-fd9e-4e94-a124-8f535c889d80\" (UID: \"f7efff85-fd9e-4e94-a124-8f535c889d80\") "
Oct 30 07:59:27.282333 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.282311    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume" pod "f7efff85-fd9e-4e94-a124-8f535c889d80" (UID: "f7efff85-fd9e-4e94-a124-8f535c889d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:59:27.282471 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.282452    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd25f\" (UniqueName: \"kubernetes.io/projected/4b48ee79-9976-42f2-8c53-5a13f51a056f-kube-api-access-cd25f\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:27.283449 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.283421    6738 scope.go:122] "RemoveContainer" containerID="c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"
Oct 30 07:59:27.283775 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:27.283747    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749\": container with ID starting with c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749 not found: ID does not exist" containerID="c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"
Oct 30 07:59:27.283891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.283779    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749"} err="failed to get container status \"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749\": rpc error: code = NotFound desc = could not find container \"c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749\": container with ID starting with c2457e3552f103d7742e23c7dbd832bf285d83cbeaf92529c516f4185b298749 not found: ID does not exist"
Oct 30 07:59:27.283891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.283796    6738 scope.go:122] "RemoveContainer" containerID="db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"
Oct 30 07:59:27.283891 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.283844    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c" pod "f7efff85-fd9e-4e94-a124-8f535c889d80" (UID: "f7efff85-fd9e-4e94-a124-8f535c889d80"). InnerVolumeSpecName "kube-api-access-2q49c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:27.294469 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.294447    6738 scope.go:122] "RemoveContainer" containerID="db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"
Oct 30 07:59:27.294755 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:27.294732    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c\": container with ID starting with db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c not found: ID does not exist" containerID="db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"
Oct 30 07:59:27.294809 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.294763    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c"} err="failed to get container status \"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c\": rpc error: code = NotFound desc = could not find container \"db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c\": container with ID starting with db78693582ccb84d49a888093c43736791f6ea75552209d35096680d4fee5b2c not found: ID does not exist"
Oct 30 07:59:27.339781 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.339743    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:59:27.346718 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.346635    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8f1df-79597cd9b8-mjj2k"]
Oct 30 07:59:27.384112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.384075    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q49c\" (UniqueName: \"kubernetes.io/projected/f7efff85-fd9e-4e94-a124-8f535c889d80-kube-api-access-2q49c\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:27.384112 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.384106    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7efff85-fd9e-4e94-a124-8f535c889d80-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:27.621483 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.621383    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:59:27.624726 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.624697    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8f1df-cbcff7f5-7vn94"]
Oct 30 07:59:27.692058 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.692024    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" path="/var/lib/kubelet/pods/4b48ee79-9976-42f2-8c53-5a13f51a056f/volumes"
Oct 30 07:59:27.692506 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.692492    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" path="/var/lib/kubelet/pods/f7efff85-fd9e-4e94-a124-8f535c889d80/volumes"
Oct 30 07:59:27.841278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:27.841236    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:29.652255 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:29.652203    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:30.945745 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:30.945711    6738 scope.go:122] "RemoveContainer" containerID="e930a88ec52476c6dfd1c5bb2c8a46086cd1bb7e5fc069a84ddab07a84faa1ea"
Oct 30 07:59:31.555016 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:31.554983    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:33.445507 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:33.445470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:35.208907 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:35.208860    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:35.473017 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:35.472925    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:37.263221 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:37.263179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:37.521578 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:37.521475    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 07:59:38.883457 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:38.883422    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:39.591920 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:39.591883    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:39.729493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:39.729460    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"]
Oct 30 07:59:39.735188 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:39.735149    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:39.741474 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:39.741448    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8f1df-t25kz"]
Oct 30 07:59:39.844697 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:39.844603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:40.225801 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:40.225766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:41.640011 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:41.639979    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:41.747373 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:41.747272    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" path="/var/lib/kubelet/pods/4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42/volumes"
Oct 30 07:59:43.507773 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:43.507736    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:45.460022 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:45.459988    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:46.449505 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:46.449467    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:46.589113 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:46.589081    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:46.816971 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:46.816886    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:59:46.817241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:46.817212    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy" containerID="cri-o://d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177" gracePeriod=30
Oct 30 07:59:46.856059 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:46.856021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:47.454638 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.454602    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"]
Oct 30 07:59:47.455389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455351    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing"
Oct 30 07:59:47.455389 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455372    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing"
Oct 30 07:59:47.455508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455400    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" containerName="revoke-keys"
Oct 30 07:59:47.455508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455405    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" containerName="revoke-keys"
Oct 30 07:59:47.455508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455456    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing"
Oct 30 07:59:47.455508 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455463    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing"
Oct 30 07:59:47.455649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455610    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b48ee79-9976-42f2-8c53-5a13f51a056f" containerName="payload-processing"
Oct 30 07:59:47.455649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455623    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7efff85-fd9e-4e94-a124-8f535c889d80" containerName="payload-pre-processing"
Oct 30 07:59:47.455649 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.455632    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1b89b7-fc3f-48aa-ae36-5317b7ab2a42" containerName="revoke-keys"
Oct 30 07:59:47.456886 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.456861    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:47.461805 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.461786    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.477241 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.477212    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"]
Oct 30 07:59:47.483493 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.483464    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:59:47.483733 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.483503    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:59:47.562397 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.558630    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.562762 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.562730    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.563062 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.563039    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpkd\" (UniqueName: \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.664288 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.664246    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.664752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.664346    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.664752 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.664444    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpkd\" (UniqueName: \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.664960 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.664937    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.666786 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.666763    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.688438 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.688403    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpkd\" (UniqueName: \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\") pod \"maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:47.777980 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:47.777888    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:48.084540 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:48.084503    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"]
Oct 30 07:59:48.084851 ip-10-0-139-124 kubenswrapper[6738]: W1030 07:59:48.084823    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe114a42_050a_4625_a659_8d8202f81728.slice/crio-982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0 WatchSource:0}: Error finding container 982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0: Status 404 returned error can't find the container with id 982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0
Oct 30 07:59:48.500357 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:48.500318    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt" event={"ID":"fe114a42-050a-4625-a659-8d8202f81728","Type":"ContainerStarted","Data":"3606e0b0fc4d4c7b50a269c1e85713178857d38b0146b2fae07522dfd3c0ca8e"}
Oct 30 07:59:48.500577 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:48.500385    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt" event={"ID":"fe114a42-050a-4625-a659-8d8202f81728","Type":"ContainerStarted","Data":"982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0"}
Oct 30 07:59:48.542194 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:48.542134    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt" podStartSLOduration=1.5421186279999999 podStartE2EDuration="1.542118628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:48.540826868 +0000 UTC m=+3748.973590761" watchObservedRunningTime="2026-10-30 07:59:48.542118628 +0000 UTC m=+3748.974882520"
Oct 30 07:59:49.413460 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:49.413421    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:49.527102 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:49.527064    6738 generic.go:356] "Generic (PLEG): container finished" podID="fe114a42-050a-4625-a659-8d8202f81728" containerID="3606e0b0fc4d4c7b50a269c1e85713178857d38b0146b2fae07522dfd3c0ca8e" exitCode=0
Oct 30 07:59:49.527315 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:49.527147    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt" event={"ID":"fe114a42-050a-4625-a659-8d8202f81728","Type":"ContainerDied","Data":"3606e0b0fc4d4c7b50a269c1e85713178857d38b0146b2fae07522dfd3c0ca8e"}
Oct 30 07:59:50.705895 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.705866    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:50.816859 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.816767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\") pod \"fe114a42-050a-4625-a659-8d8202f81728\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") "
Oct 30 07:59:50.817038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.816954    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\") pod \"fe114a42-050a-4625-a659-8d8202f81728\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") "
Oct 30 07:59:50.817038 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.817020    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\" (UniqueName: \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\") pod \"fe114a42-050a-4625-a659-8d8202f81728\" (UID: \"fe114a42-050a-4625-a659-8d8202f81728\") "
Oct 30 07:59:50.817356 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.817333    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca" pod "fe114a42-050a-4625-a659-8d8202f81728" (UID: "fe114a42-050a-4625-a659-8d8202f81728"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:59:50.818903 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.818877    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca" pod "fe114a42-050a-4625-a659-8d8202f81728" (UID: "fe114a42-050a-4625-a659-8d8202f81728"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:59:50.819142 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.819120    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd" pod "fe114a42-050a-4625-a659-8d8202f81728" (UID: "fe114a42-050a-4625-a659-8d8202f81728"). InnerVolumeSpecName "kube-api-access-5bpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:50.918432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.918392    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe114a42-050a-4625-a659-8d8202f81728-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:50.918432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.918419    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe114a42-050a-4625-a659-8d8202f81728-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:50.918432 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:50.918432    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpkd\" (UniqueName: \"kubernetes.io/projected/fe114a42-050a-4625-a659-8d8202f81728-kube-api-access-5bpkd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:51.449758 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:51.449717    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:51.543957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:51.543920    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"
Oct 30 07:59:51.543957 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:51.543950    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt" event={"ID":"fe114a42-050a-4625-a659-8d8202f81728","Type":"ContainerDied","Data":"982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0"}
Oct 30 07:59:51.544183 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:51.544004    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982b76d14db9a4ed81ee4b6efe8dedc08645760ece675ea6ced917e040f520f0"
Oct 30 07:59:52.104562 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.104488    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:59:52.244425 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244384    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.244662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244499    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.244662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244578    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.244662 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244631    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.244882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244756    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.244882 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244856    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.245005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244905    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.245005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244833    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:59:52.245005 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.244910    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:59:52.245193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245011    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.245193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245074    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\") pod \"eac308aa-bb24-44b4-8350-a94926d276b9\" (UID: \"eac308aa-bb24-44b4-8350-a94926d276b9\") "
Oct 30 07:59:52.245193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245102    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:59:52.245193 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245139    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:59:52.245802 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245773    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac308aa-bb24-44b4-8350-a94926d276b9-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.245925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245804    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.245925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245819    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.245925 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245832    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.246066 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.245924    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:59:52.248050 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.248020    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:59:52.248216 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.248196    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "kube-api-access-v89fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:52.248452 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.248430    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:59:52.249390 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.249371    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token" pod "eac308aa-bb24-44b4-8350-a94926d276b9" (UID: "eac308aa-bb24-44b4-8350-a94926d276b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:52.347386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.347352    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.347386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.347386    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac308aa-bb24-44b4-8350-a94926d276b9-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.347386 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.347401    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.347674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.347415    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac308aa-bb24-44b4-8350-a94926d276b9-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.347674 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.347428    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89fp\" (UniqueName: \"kubernetes.io/projected/eac308aa-bb24-44b4-8350-a94926d276b9-kube-api-access-v89fp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 07:59:52.553358 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.553320    6738 generic.go:356] "Generic (PLEG): container finished" podID="eac308aa-bb24-44b4-8350-a94926d276b9" containerID="d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177" exitCode=0
Oct 30 07:59:52.553551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.553363    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"
Oct 30 07:59:52.553551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.553398    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerDied","Data":"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"}
Oct 30 07:59:52.553551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.553480    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d" event={"ID":"eac308aa-bb24-44b4-8350-a94926d276b9","Type":"ContainerDied","Data":"addfb5c12bf0967602bfbc82d69c5d9303060b1796b5e61181013cff82f329db"}
Oct 30 07:59:52.553551 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.553502    6738 scope.go:122] "RemoveContainer" containerID="d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"
Oct 30 07:59:52.566055 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.566030    6738 scope.go:122] "RemoveContainer" containerID="d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"
Oct 30 07:59:52.577407 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.577383    6738 scope.go:122] "RemoveContainer" containerID="d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"
Oct 30 07:59:52.577721 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:52.577699    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177\": container with ID starting with d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177 not found: ID does not exist" containerID="d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"
Oct 30 07:59:52.577769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.577731    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177"} err="failed to get container status \"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177\": rpc error: code = NotFound desc = could not find container \"d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177\": container with ID starting with d9511088bc02e7978b53ba6c67edd1e8fcc6c73ba76e0b038d4763d4a6e94177 not found: ID does not exist"
Oct 30 07:59:52.577769 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.577751    6738 scope.go:122] "RemoveContainer" containerID="d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"
Oct 30 07:59:52.578051 ip-10-0-139-124 kubenswrapper[6738]: E1030 07:59:52.578029    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de\": container with ID starting with d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de not found: ID does not exist" containerID="d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"
Oct 30 07:59:52.578124 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.578056    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de"} err="failed to get container status \"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de\": rpc error: code = NotFound desc = could not find container \"d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de\": container with ID starting with d822b1364bb8d618768b628a56ea6bbb0d4188e0342d203855e98fd87ccf35de not found: ID does not exist"
Oct 30 07:59:52.609816 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.609773    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:59:52.621979 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:52.621944    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8f1df-openshift-default-6bd8db459c-kxf5d"]
Oct 30 07:59:53.204408 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:53.204373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:53.692465 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:53.692429    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" path="/var/lib/kubelet/pods/eac308aa-bb24-44b4-8350-a94926d276b9/volumes"
Oct 30 07:59:54.984657 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:54.984623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:56.996264 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:56.996219    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:57.932304 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:57.932252    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 07:59:58.110640 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:58.110603    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:58.236416 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:58.236238    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 07:59:59.067278 ip-10-0-139-124 kubenswrapper[6738]: I1030 07:59:59.067242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:00.158411 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.158376    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:00.159134 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159120    6738 cpu_manager.go:414] "containerMap: removing container" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.159201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159136    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.159201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159150    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fe114a42-050a-4625-a659-8d8202f81728" containerName="revoke-keys"
Oct 30 08:00:00.159201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159155    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe114a42-050a-4625-a659-8d8202f81728" containerName="revoke-keys"
Oct 30 08:00:00.159201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159163    6738 cpu_manager.go:414] "containerMap: removing container" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.159201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159169    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.159389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159353    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.159389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159368    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe114a42-050a-4625-a659-8d8202f81728" containerName="revoke-keys"
Oct 30 08:00:00.159389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.159376    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac308aa-bb24-44b4-8350-a94926d276b9" containerName="istio-proxy"
Oct 30 08:00:00.163555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.163533    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.172609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.172583    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:00.180934 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.180908    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:00.181090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.181019    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.185943 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.185915    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:00.186096 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.186014    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.190603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.190576    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.197871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.197834    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:00:00.198066 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.198046    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:00.221543 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.215954    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:00.241031 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.240063    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:00.242653 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.242604    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:00.248228 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.248189    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:00.248590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.248563    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.257330 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.253787    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:00.257330 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.253916    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.278454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.278274    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:00.283967 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.283936    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.302637 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.302606    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:00.302905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.302865    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbk7d\" (UniqueName: \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.303140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303029    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgt8\" (UniqueName: \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.303140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303079    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.303277 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303141    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvz4\" (UniqueName: \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.303402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303249    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b488f\" (UniqueName: \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.303495 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303425    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.303609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303589    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.303689 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303624    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.303752 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.303812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303760    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvs4k\" (UniqueName: \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.303872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303809    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpd9\" (UniqueName: \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.304021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.303939    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.312751 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.312709    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:00.327000 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.326967    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:00.331957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.331934    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.333469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.333441    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:00.343161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.343136    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.345954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.345925    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:00.397817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.395645    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:00.404686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.404619    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:00.409018 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.408879    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbk7d\" (UniqueName: \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.409430 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.409338    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgt8\" (UniqueName: \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.409628 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.409610    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.409772 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.409754    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvz4\" (UniqueName: \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.410011 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.409913    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.410226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.410205    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b488f\" (UniqueName: \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.410481 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.410359    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.410776 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.410642    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlm9\" (UniqueName: \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.411001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.410936    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.411217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411142    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.411401 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411384    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.411642 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411623    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs4k\" (UniqueName: \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.411821 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411803    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmdv\" (UniqueName: \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.412004 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411970    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpd9\" (UniqueName: \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.412553 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.412526    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsdc\" (UniqueName: \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.412732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.412717    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.412852 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.412834    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.412978 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.412964    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.414111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.413619    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.414111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.411694    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.414111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.412773    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.414551 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.414528    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.414930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.414811    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.414930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.414842    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.415886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.415855    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:00.426097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.425931    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:00.427723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.426821    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.432162 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.432138    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.457435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.457389    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b488f\" (UniqueName: \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.463367 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.457397    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:00.463583 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.463569    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:00.463688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.460689    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvz4\" (UniqueName: \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\") pod \"maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.463794 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.463703    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpd9\" (UniqueName: \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.478634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.478602    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbk7d\" (UniqueName: \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\") pod \"maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.483648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.483417    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:00.500898 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.500632    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:00.508353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.506542    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:00.509306 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.509220    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvs4k\" (UniqueName: \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.512400    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgt8\" (UniqueName: \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514458    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514499    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514537    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpw8j\" (UniqueName: \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514569    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514622    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdnm\" (UniqueName: \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514700    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514801    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlm9\" (UniqueName: \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514928    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmdv\" (UniqueName: \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.514974    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsdc\" (UniqueName: \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.516042    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.522017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.516701    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.524235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.522092    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:00.524655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.524594    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:00.525617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.525556    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.526264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.526177    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.540604 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.538547    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:00.551055 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.549658    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:00.551055 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.549782    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.551589 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.551503    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:00.552634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.552573    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlm9\" (UniqueName: \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.555609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.555587    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsdc\" (UniqueName: \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.562026 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.561939    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmdv\" (UniqueName: \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.569193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.569164    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:00.581107 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.581075    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:00.613112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.613077    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:00.618266 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618160    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.618266 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618241    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.618414 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618356    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.618414 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618394    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rql5\" (UniqueName: \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618429    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpw8j\" (UniqueName: \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618452    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.618528 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618516    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdnm\" (UniqueName: \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.618738 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.618561    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fd4p\" (UniqueName: \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.620887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.619908    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.620887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.620835    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.655167 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.654929    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:00.657937 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.656975    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpw8j\" (UniqueName: \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.668634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.668364    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:00.674889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.674837    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdnm\" (UniqueName: \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\") pod \"maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.720343    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.720388    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rql5\" (UniqueName: \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.720504    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fd4p\" (UniqueName: \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.720758    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.721575    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.722829 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.722536    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.763278 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.760122    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fd4p\" (UniqueName: \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\") pod \"maas-api-key-cleanup-29889120-bwz8n\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:00.778180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.777766    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rql5\" (UniqueName: \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.811022 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.810730    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:00.825676 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.825642    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:00.892385 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.891924    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:00.893036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.893001    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:00.904513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:00.904484    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:01.100810 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.100776    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:01.140203 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.140157    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:01.312057 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.312020    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:01.319659 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.319630    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:01.344029 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.343992    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:01.374838 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.374687    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:01.473442 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.473334    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:01.490665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.490582    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:01.524096 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.523902    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:01.530021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.529749    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:01.538039 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.537700    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:01.554915 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.554885    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:01.563338 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.563309    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:01.704492 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.704442    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerStarted","Data":"83fbe1dd8bf6b0d5dccfe210c51308c6cba8d9770faf5b2b69dba5a4f4d24615"}
Oct 30 08:00:01.707578 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.707521    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" event={"ID":"147e81ec-5cb1-4b49-a878-e5f170457614","Type":"ContainerStarted","Data":"d177647a92cf8db936a88c7d13ccead468843520039a7c38778ab59903ae4255"}
Oct 30 08:00:01.710578 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.710543    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerStarted","Data":"e350444d6067167c7c9a6b93d50549ee8423d7c76196b916eacbe081c3401385"}
Oct 30 08:00:01.713062 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.713031    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerStarted","Data":"e15855becd5c2f6f1928618e871e6faefe32d17838c133e8fa838b8c15a4dd1a"}
Oct 30 08:00:01.715214 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.715189    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerStarted","Data":"86f1f338fa19cc590e67f59695a830af213236accde2fe293386adc610a5aba0"}
Oct 30 08:00:01.721227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.721196    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerStarted","Data":"a6242840df71b3e8a24ddb1e8bbd11c4eb970c7ce4e78084fb22b97fc2a8b434"}
Oct 30 08:00:01.722763 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.722740    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerStarted","Data":"33f8ffcc3bd533bcba9f88196fbbf7e7dc5d1ad743dcc63da45d1d82ba943020"}
Oct 30 08:00:01.724141 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.724120    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerStarted","Data":"5a37c4eda429bfff07af1f6b569157af1fc669fc19ce92c2e2fd9014770fd0a8"}
Oct 30 08:00:01.724239 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.724158    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerStarted","Data":"63741188ca8dd98a7983c02776f134b6c03904fba833f2a5200fad7924f712e3"}
Oct 30 08:00:01.726335 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.726235    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerStarted","Data":"fa6bc9b4ce49991703069b78cde614f3b638662a58cb96fad462cd2c15fc0ef4"}
Oct 30 08:00:01.727764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.727741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" event={"ID":"6d5b65bd-7e5e-4d28-988a-dd2035b41572","Type":"ContainerStarted","Data":"e47da4a934c4e39bed2ae4a58d15b1e5053eea581c94a38034be95a088ca5384"}
Oct 30 08:00:01.728919 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.728897    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerStarted","Data":"64165c97cb009c47c93e04f927efc3d4c006fc35f25ec27ce167fe6f8c7a8fbb"}
Oct 30 08:00:01.730224 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.730201    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerStarted","Data":"4eedf640e2eba51bcd38837a6461fd8deac3fceb7ec6ab487d29fccd719ea57d"}
Oct 30 08:00:01.731484 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.731463    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerStarted","Data":"cd2db2fec7aef880cf3cbe8149ca1bd534e3641fd0f438e184615da54e784a7c"}
Oct 30 08:00:01.780214 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:01.780147    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" podStartSLOduration=1.780125955 podStartE2EDuration="1.780125955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.775393119 +0000 UTC m=+3762.208157016" watchObservedRunningTime="2026-10-30 08:00:01.780125955 +0000 UTC m=+3762.212889854"
Oct 30 08:00:02.744129 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.744089    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerStarted","Data":"399594df0283feca5bc259ba69384f5aebeb2424d7d3827135c2d562f90b1a9f"}
Oct 30 08:00:02.746189 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.746161    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerStarted","Data":"98acd070e78aff257af32f9c40d45287671e249ce3ab1e98fafbfe7f81203548"}
Oct 30 08:00:02.748118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.748087    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerStarted","Data":"e15142baa8fa3a8f19ab143abc2a6ab90fba3bf6a6744a9773c1ba902e400556"}
Oct 30 08:00:02.749881 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.749854    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerStarted","Data":"4521d8bcdd844ef11c2772860304262ace0a0f144b7a1764d1f3787511b2eb8e"}
Oct 30 08:00:02.751564 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.751542    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerStarted","Data":"c8de699e3b05b71788b4b5125caafdcc2ffacc5ff17c55158bce8e6b5e7481b2"}
Oct 30 08:00:02.753201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.753177    6738 generic.go:356] "Generic (PLEG): container finished" podID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerID="5a37c4eda429bfff07af1f6b569157af1fc669fc19ce92c2e2fd9014770fd0a8" exitCode=60
Oct 30 08:00:02.753349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.753254    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerDied","Data":"5a37c4eda429bfff07af1f6b569157af1fc669fc19ce92c2e2fd9014770fd0a8"}
Oct 30 08:00:02.753517 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.753500    6738 scope.go:122] "RemoveContainer" containerID="5a37c4eda429bfff07af1f6b569157af1fc669fc19ce92c2e2fd9014770fd0a8"
Oct 30 08:00:02.755109 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.755058    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerStarted","Data":"6bfea46be9e3de2f7971d0e283811bef913a6f470666eb946df465a8a5704678"}
Oct 30 08:00:02.757267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.757245    6738 generic.go:356] "Generic (PLEG): container finished" podID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerID="5ba06dbe93345b690afa17da2c6ea6a5152bf38b6ac8ce5542d06b65bce7315d" exitCode=60
Oct 30 08:00:02.757383 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.757317    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" event={"ID":"6d5b65bd-7e5e-4d28-988a-dd2035b41572","Type":"ContainerDied","Data":"5ba06dbe93345b690afa17da2c6ea6a5152bf38b6ac8ce5542d06b65bce7315d"}
Oct 30 08:00:02.757554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.757530    6738 scope.go:122] "RemoveContainer" containerID="5ba06dbe93345b690afa17da2c6ea6a5152bf38b6ac8ce5542d06b65bce7315d"
Oct 30 08:00:02.759133 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.759112    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerStarted","Data":"74668cb0167ab28946388d5681abd874d539846a07743b94994bd7b95bd45540"}
Oct 30 08:00:02.760851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.760829    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerStarted","Data":"4d58438a9141593cc90963546a97f9e3c296d313de1ea34a0fd75eca9561f6a0"}
Oct 30 08:00:02.762490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.762469    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerStarted","Data":"491c743e142b705f4422bc5811b96cc169d28adfa1b00bd70621b6c951b8754b"}
Oct 30 08:00:02.764403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.764377    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerStarted","Data":"e8f67b42ae8d0038ca2650a9fddbd06bedcae160f7817337e52ba696e2ff3571"}
Oct 30 08:00:02.766370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.766344    6738 generic.go:356] "Generic (PLEG): container finished" podID="147e81ec-5cb1-4b49-a878-e5f170457614" containerID="f63e92998b3613dbd0e3bea8f38c19bc96e2d25507b6a59a70816fe01bd1e2a3" exitCode=60
Oct 30 08:00:02.766547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.766411    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" event={"ID":"147e81ec-5cb1-4b49-a878-e5f170457614","Type":"ContainerDied","Data":"f63e92998b3613dbd0e3bea8f38c19bc96e2d25507b6a59a70816fe01bd1e2a3"}
Oct 30 08:00:02.766659 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.766639    6738 scope.go:122] "RemoveContainer" containerID="f63e92998b3613dbd0e3bea8f38c19bc96e2d25507b6a59a70816fe01bd1e2a3"
Oct 30 08:00:02.801424 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.801379    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" podStartSLOduration=2.801365255 podStartE2EDuration="2.801365255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.795757836 +0000 UTC m=+3763.228521730" watchObservedRunningTime="2026-10-30 08:00:02.801365255 +0000 UTC m=+3763.234129147"
Oct 30 08:00:02.918428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.918372    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" podStartSLOduration=2.918357132 podStartE2EDuration="2.918357132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.916855609 +0000 UTC m=+3763.349619501" watchObservedRunningTime="2026-10-30 08:00:02.918357132 +0000 UTC m=+3763.351121025"
Oct 30 08:00:02.933570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.933512    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing" containerID="cri-o://40284472999dff81a91eb3d1f79269d0c16cc228f17c1abfb1978e0cb6a1d6ba" gracePeriod=25
Oct 30 08:00:02.973935 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:02.973878    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" podStartSLOduration=2.973858445 podStartE2EDuration="2.973858445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.969933009 +0000 UTC m=+3763.402696903" watchObservedRunningTime="2026-10-30 08:00:02.973858445 +0000 UTC m=+3763.406622339"
Oct 30 08:00:03.039726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.039660    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" podStartSLOduration=3.039638092 podStartE2EDuration="3.039638092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.0361334 +0000 UTC m=+3763.468897293" watchObservedRunningTime="2026-10-30 08:00:03.039638092 +0000 UTC m=+3763.472401983"
Oct 30 08:00:03.089784 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.089724    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" podStartSLOduration=3.089711184 podStartE2EDuration="3.089711184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.085646971 +0000 UTC m=+3763.518410866" watchObservedRunningTime="2026-10-30 08:00:03.089711184 +0000 UTC m=+3763.522475076"
Oct 30 08:00:03.137834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.137776    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" podStartSLOduration=3.137758938 podStartE2EDuration="3.137758938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.1372396 +0000 UTC m=+3763.570003494" watchObservedRunningTime="2026-10-30 08:00:03.137758938 +0000 UTC m=+3763.570522830"
Oct 30 08:00:03.181411 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.181363    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" podStartSLOduration=3.181348972 podStartE2EDuration="3.181348972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.180641812 +0000 UTC m=+3763.613405704" watchObservedRunningTime="2026-10-30 08:00:03.181348972 +0000 UTC m=+3763.614112865"
Oct 30 08:00:03.272902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.272790    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" podStartSLOduration=3.272775748 podStartE2EDuration="3.272775748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.268640382 +0000 UTC m=+3763.701404281" watchObservedRunningTime="2026-10-30 08:00:03.272775748 +0000 UTC m=+3763.705539642"
Oct 30 08:00:03.356242 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.356186    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" podStartSLOduration=3.356171574 podStartE2EDuration="3.356171574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.352627232 +0000 UTC m=+3763.785391125" watchObservedRunningTime="2026-10-30 08:00:03.356171574 +0000 UTC m=+3763.788935466"
Oct 30 08:00:03.397040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.396987    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" podStartSLOduration=3.396970292 podStartE2EDuration="3.396970292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.394434644 +0000 UTC m=+3763.827198537" watchObservedRunningTime="2026-10-30 08:00:03.396970292 +0000 UTC m=+3763.829734186"
Oct 30 08:00:03.668856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.668816    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:03.782712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.782672    6738 generic.go:356] "Generic (PLEG): container finished" podID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerID="399594df0283feca5bc259ba69384f5aebeb2424d7d3827135c2d562f90b1a9f" exitCode=60
Oct 30 08:00:03.783153 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.782767    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerDied","Data":"399594df0283feca5bc259ba69384f5aebeb2424d7d3827135c2d562f90b1a9f"}
Oct 30 08:00:03.783153 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.783133    6738 scope.go:122] "RemoveContainer" containerID="399594df0283feca5bc259ba69384f5aebeb2424d7d3827135c2d562f90b1a9f"
Oct 30 08:00:03.785339 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.785246    6738 generic.go:356] "Generic (PLEG): container finished" podID="920e7877-1932-4a3a-a845-b61357d5e804" containerID="98acd070e78aff257af32f9c40d45287671e249ce3ab1e98fafbfe7f81203548" exitCode=60
Oct 30 08:00:03.785473 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.785447    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerDied","Data":"98acd070e78aff257af32f9c40d45287671e249ce3ab1e98fafbfe7f81203548"}
Oct 30 08:00:03.785745 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.785713    6738 scope.go:122] "RemoveContainer" containerID="98acd070e78aff257af32f9c40d45287671e249ce3ab1e98fafbfe7f81203548"
Oct 30 08:00:03.787985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.787853    6738 generic.go:356] "Generic (PLEG): container finished" podID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerID="e15142baa8fa3a8f19ab143abc2a6ab90fba3bf6a6744a9773c1ba902e400556" exitCode=60
Oct 30 08:00:03.787985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.787880    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerDied","Data":"e15142baa8fa3a8f19ab143abc2a6ab90fba3bf6a6744a9773c1ba902e400556"}
Oct 30 08:00:03.788181 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.788169    6738 scope.go:122] "RemoveContainer" containerID="e15142baa8fa3a8f19ab143abc2a6ab90fba3bf6a6744a9773c1ba902e400556"
Oct 30 08:00:03.789931 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.789914    6738 generic.go:356] "Generic (PLEG): container finished" podID="61865956-c60b-471e-a066-33cbee6cd28c" containerID="4521d8bcdd844ef11c2772860304262ace0a0f144b7a1764d1f3787511b2eb8e" exitCode=60
Oct 30 08:00:03.790031 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.790001    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerDied","Data":"4521d8bcdd844ef11c2772860304262ace0a0f144b7a1764d1f3787511b2eb8e"}
Oct 30 08:00:03.790200 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.790186    6738 scope.go:122] "RemoveContainer" containerID="4521d8bcdd844ef11c2772860304262ace0a0f144b7a1764d1f3787511b2eb8e"
Oct 30 08:00:03.791998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.791978    6738 generic.go:356] "Generic (PLEG): container finished" podID="57826627-652f-4b1b-a66a-84d6075e13a4" containerID="c8de699e3b05b71788b4b5125caafdcc2ffacc5ff17c55158bce8e6b5e7481b2" exitCode=60
Oct 30 08:00:03.792107 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.792029    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerDied","Data":"c8de699e3b05b71788b4b5125caafdcc2ffacc5ff17c55158bce8e6b5e7481b2"}
Oct 30 08:00:03.792359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.792341    6738 scope.go:122] "RemoveContainer" containerID="c8de699e3b05b71788b4b5125caafdcc2ffacc5ff17c55158bce8e6b5e7481b2"
Oct 30 08:00:03.794196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.794174    6738 generic.go:356] "Generic (PLEG): container finished" podID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerID="6bfea46be9e3de2f7971d0e283811bef913a6f470666eb946df465a8a5704678" exitCode=60
Oct 30 08:00:03.794327 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.794266    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerDied","Data":"6bfea46be9e3de2f7971d0e283811bef913a6f470666eb946df465a8a5704678"}
Oct 30 08:00:03.794566 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.794547    6738 scope.go:122] "RemoveContainer" containerID="6bfea46be9e3de2f7971d0e283811bef913a6f470666eb946df465a8a5704678"
Oct 30 08:00:03.801028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.801006    6738 generic.go:356] "Generic (PLEG): container finished" podID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerID="74668cb0167ab28946388d5681abd874d539846a07743b94994bd7b95bd45540" exitCode=60
Oct 30 08:00:03.801270 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.801253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerDied","Data":"74668cb0167ab28946388d5681abd874d539846a07743b94994bd7b95bd45540"}
Oct 30 08:00:03.801456 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.801440    6738 scope.go:122] "RemoveContainer" containerID="74668cb0167ab28946388d5681abd874d539846a07743b94994bd7b95bd45540"
Oct 30 08:00:03.803847 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.803823    6738 generic.go:356] "Generic (PLEG): container finished" podID="c5232edb-fed4-493e-96f9-077bba327cca" containerID="4d58438a9141593cc90963546a97f9e3c296d313de1ea34a0fd75eca9561f6a0" exitCode=60
Oct 30 08:00:03.803957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.803899    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerDied","Data":"4d58438a9141593cc90963546a97f9e3c296d313de1ea34a0fd75eca9561f6a0"}
Oct 30 08:00:03.804100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.804076    6738 scope.go:122] "RemoveContainer" containerID="4d58438a9141593cc90963546a97f9e3c296d313de1ea34a0fd75eca9561f6a0"
Oct 30 08:00:03.807778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.806487    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerID="491c743e142b705f4422bc5811b96cc169d28adfa1b00bd70621b6c951b8754b" exitCode=60
Oct 30 08:00:03.807778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.806574    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerDied","Data":"491c743e142b705f4422bc5811b96cc169d28adfa1b00bd70621b6c951b8754b"}
Oct 30 08:00:03.809228 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.808459    6738 scope.go:122] "RemoveContainer" containerID="491c743e142b705f4422bc5811b96cc169d28adfa1b00bd70621b6c951b8754b"
Oct 30 08:00:03.812117 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.812092    6738 generic.go:356] "Generic (PLEG): container finished" podID="199c86b9-53fc-4184-8007-8639510fe71a" containerID="e8f67b42ae8d0038ca2650a9fddbd06bedcae160f7817337e52ba696e2ff3571" exitCode=60
Oct 30 08:00:03.812530 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.812479    6738 scope.go:122] "RemoveContainer" containerID="e8f67b42ae8d0038ca2650a9fddbd06bedcae160f7817337e52ba696e2ff3571"
Oct 30 08:00:03.812670 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:03.812650    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerDied","Data":"e8f67b42ae8d0038ca2650a9fddbd06bedcae160f7817337e52ba696e2ff3571"}
Oct 30 08:00:04.823490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.823393    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerStarted","Data":"9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"}
Oct 30 08:00:04.825794 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.825763    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerStarted","Data":"b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"}
Oct 30 08:00:04.828875 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.828842    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerStarted","Data":"c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"}
Oct 30 08:00:04.831438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.831414    6738 generic.go:356] "Generic (PLEG): container finished" podID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerID="5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80" exitCode=60
Oct 30 08:00:04.831575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.831491    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerDied","Data":"5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80"}
Oct 30 08:00:04.831575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.831543    6738 scope.go:122] "RemoveContainer" containerID="5a37c4eda429bfff07af1f6b569157af1fc669fc19ce92c2e2fd9014770fd0a8"
Oct 30 08:00:04.831766 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.831747    6738 scope.go:122] "RemoveContainer" containerID="5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80"
Oct 30 08:00:04.832047 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:04.832022    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf_odh-ai-gateway-infra(ecd7ba37-da03-4528-a350-c4973aa9f9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4"
Oct 30 08:00:04.833698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.833675    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerStarted","Data":"2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"}
Oct 30 08:00:04.835715 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.835694    6738 generic.go:356] "Generic (PLEG): container finished" podID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerID="af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211" exitCode=60
Oct 30 08:00:04.835832 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.835763    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" event={"ID":"6d5b65bd-7e5e-4d28-988a-dd2035b41572","Type":"ContainerDied","Data":"af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211"}
Oct 30 08:00:04.836075 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.836058    6738 scope.go:122] "RemoveContainer" containerID="af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211"
Oct 30 08:00:04.836329 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:04.836307    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4_odh-ai-gateway-infra(6d5b65bd-7e5e-4d28-988a-dd2035b41572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572"
Oct 30 08:00:04.838169 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.838143    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerStarted","Data":"2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"}
Oct 30 08:00:04.840859 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.840837    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerStarted","Data":"f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"}
Oct 30 08:00:04.842765 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.842744    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerStarted","Data":"94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"}
Oct 30 08:00:04.844844 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.844814    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerStarted","Data":"c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"}
Oct 30 08:00:04.846954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.846933    6738 generic.go:356] "Generic (PLEG): container finished" podID="147e81ec-5cb1-4b49-a878-e5f170457614" containerID="c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d" exitCode=60
Oct 30 08:00:04.847077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.847060    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" event={"ID":"147e81ec-5cb1-4b49-a878-e5f170457614","Type":"ContainerDied","Data":"c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d"}
Oct 30 08:00:04.847274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.847241    6738 scope.go:122] "RemoveContainer" containerID="c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d"
Oct 30 08:00:04.847566 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:04.847536    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps_odh-ai-gateway-infra(147e81ec-5cb1-4b49-a878-e5f170457614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" podUID="147e81ec-5cb1-4b49-a878-e5f170457614"
Oct 30 08:00:04.849199 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.849182    6738 scope.go:122] "RemoveContainer" containerID="5ba06dbe93345b690afa17da2c6ea6a5152bf38b6ac8ce5542d06b65bce7315d"
Oct 30 08:00:04.849454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.849427    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerStarted","Data":"4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"}
Oct 30 08:00:04.851388 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.851369    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerStarted","Data":"6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"}
Oct 30 08:00:04.877269 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:04.877171    6738 scope.go:122] "RemoveContainer" containerID="f63e92998b3613dbd0e3bea8f38c19bc96e2d25507b6a59a70816fe01bd1e2a3"
Oct 30 08:00:05.633761 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.633723    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:05.861978 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.861934    6738 generic.go:356] "Generic (PLEG): container finished" podID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerID="4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e" exitCode=60
Oct 30 08:00:05.862426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.862016    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerDied","Data":"4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"}
Oct 30 08:00:05.862426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.862074    6738 scope.go:122] "RemoveContainer" containerID="399594df0283feca5bc259ba69384f5aebeb2424d7d3827135c2d562f90b1a9f"
Oct 30 08:00:05.862426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.862348    6738 scope.go:122] "RemoveContainer" containerID="4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"
Oct 30 08:00:05.862738 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.862716    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-bwz8n_odh-ai-gateway-infra(295bf5fa-d510-4779-84c5-fcbb3006bd62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62"
Oct 30 08:00:05.864244 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.864224    6738 generic.go:356] "Generic (PLEG): container finished" podID="920e7877-1932-4a3a-a845-b61357d5e804" containerID="6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659" exitCode=60
Oct 30 08:00:05.864389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.864340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerDied","Data":"6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"}
Oct 30 08:00:05.864592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.864575    6738 scope.go:122] "RemoveContainer" containerID="6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"
Oct 30 08:00:05.864975 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.864951    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc_odh-ai-gateway-infra(920e7877-1932-4a3a-a845-b61357d5e804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" podUID="920e7877-1932-4a3a-a845-b61357d5e804"
Oct 30 08:00:05.866523 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.866501    6738 generic.go:356] "Generic (PLEG): container finished" podID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerID="9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c" exitCode=60
Oct 30 08:00:05.866630 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.866578    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerDied","Data":"9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"}
Oct 30 08:00:05.867180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.867157    6738 scope.go:122] "RemoveContainer" containerID="9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"
Oct 30 08:00:05.867563 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.867541    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j_odh-ai-gateway-infra(0d904f9f-9c80-4d8d-a0be-13c1b179afe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1"
Oct 30 08:00:05.868837 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.868814    6738 generic.go:356] "Generic (PLEG): container finished" podID="61865956-c60b-471e-a066-33cbee6cd28c" containerID="b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11" exitCode=60
Oct 30 08:00:05.868929 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.868873    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerDied","Data":"b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"}
Oct 30 08:00:05.869125 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.869109    6738 scope.go:122] "RemoveContainer" containerID="b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"
Oct 30 08:00:05.869470 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.869446    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w_odh-ai-gateway-infra(61865956-c60b-471e-a066-33cbee6cd28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" podUID="61865956-c60b-471e-a066-33cbee6cd28c"
Oct 30 08:00:05.871209 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.871185    6738 generic.go:356] "Generic (PLEG): container finished" podID="57826627-652f-4b1b-a66a-84d6075e13a4" containerID="c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824" exitCode=60
Oct 30 08:00:05.871345 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.871271    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerDied","Data":"c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"}
Oct 30 08:00:05.871666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.871644    6738 scope.go:122] "RemoveContainer" containerID="c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"
Oct 30 08:00:05.871957 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.871932    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd_odh-ai-gateway-infra(57826627-652f-4b1b-a66a-84d6075e13a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" podUID="57826627-652f-4b1b-a66a-84d6075e13a4"
Oct 30 08:00:05.873703 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.873681    6738 scope.go:122] "RemoveContainer" containerID="5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80"
Oct 30 08:00:05.873983 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.873962    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf_odh-ai-gateway-infra(ecd7ba37-da03-4528-a350-c4973aa9f9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4"
Oct 30 08:00:05.875526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.875505    6738 generic.go:356] "Generic (PLEG): container finished" podID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerID="2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10" exitCode=60
Oct 30 08:00:05.875627 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.875531    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerDied","Data":"2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"}
Oct 30 08:00:05.875803 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.875785    6738 scope.go:122] "RemoveContainer" containerID="2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"
Oct 30 08:00:05.876099 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.876073    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd_odh-ai-gateway-infra(a48710cf-4d3c-4d1c-a0dd-893da16b8bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6"
Oct 30 08:00:05.877576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.877559    6738 scope.go:122] "RemoveContainer" containerID="af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211"
Oct 30 08:00:05.877844 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.877812    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4_odh-ai-gateway-infra(6d5b65bd-7e5e-4d28-988a-dd2035b41572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572"
Oct 30 08:00:05.880685 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.880665    6738 scope.go:122] "RemoveContainer" containerID="98acd070e78aff257af32f9c40d45287671e249ce3ab1e98fafbfe7f81203548"
Oct 30 08:00:05.883946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.883890    6738 generic.go:356] "Generic (PLEG): container finished" podID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerID="2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166" exitCode=60
Oct 30 08:00:05.884049 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.884000    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerDied","Data":"2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"}
Oct 30 08:00:05.884382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.884363    6738 scope.go:122] "RemoveContainer" containerID="2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"
Oct 30 08:00:05.884701 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.884676    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm_odh-ai-gateway-infra(51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"
Oct 30 08:00:05.886880 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.886857    6738 generic.go:356] "Generic (PLEG): container finished" podID="c5232edb-fed4-493e-96f9-077bba327cca" containerID="f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb" exitCode=60
Oct 30 08:00:05.886993 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.886930    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerDied","Data":"f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"}
Oct 30 08:00:05.887227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.887207    6738 scope.go:122] "RemoveContainer" containerID="f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"
Oct 30 08:00:05.887594 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.887570    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl_odh-ai-gateway-infra(c5232edb-fed4-493e-96f9-077bba327cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" podUID="c5232edb-fed4-493e-96f9-077bba327cca"
Oct 30 08:00:05.889178 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.889154    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerID="94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a" exitCode=60
Oct 30 08:00:05.889315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.889236    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerDied","Data":"94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"}
Oct 30 08:00:05.889470 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.889455    6738 scope.go:122] "RemoveContainer" containerID="94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"
Oct 30 08:00:05.889724 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.889700    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl_odh-ai-gateway-infra(f8972292-ee0a-4d7a-be41-fb9d32dd4043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043"
Oct 30 08:00:05.891080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.891056    6738 generic.go:356] "Generic (PLEG): container finished" podID="199c86b9-53fc-4184-8007-8639510fe71a" containerID="c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92" exitCode=60
Oct 30 08:00:05.891196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.891131    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerDied","Data":"c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"}
Oct 30 08:00:05.891349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.891333    6738 scope.go:122] "RemoveContainer" containerID="c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"
Oct 30 08:00:05.891616 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.891595    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5_odh-ai-gateway-infra(199c86b9-53fc-4184-8007-8639510fe71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" podUID="199c86b9-53fc-4184-8007-8639510fe71a"
Oct 30 08:00:05.893202 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.893181    6738 scope.go:122] "RemoveContainer" containerID="c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d"
Oct 30 08:00:05.893508 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:05.893485    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps_odh-ai-gateway-infra(147e81ec-5cb1-4b49-a878-e5f170457614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" podUID="147e81ec-5cb1-4b49-a878-e5f170457614"
Oct 30 08:00:05.898574 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.898552    6738 scope.go:122] "RemoveContainer" containerID="e15142baa8fa3a8f19ab143abc2a6ab90fba3bf6a6744a9773c1ba902e400556"
Oct 30 08:00:05.916005 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.915980    6738 scope.go:122] "RemoveContainer" containerID="4521d8bcdd844ef11c2772860304262ace0a0f144b7a1764d1f3787511b2eb8e"
Oct 30 08:00:05.931115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.931093    6738 scope.go:122] "RemoveContainer" containerID="c8de699e3b05b71788b4b5125caafdcc2ffacc5ff17c55158bce8e6b5e7481b2"
Oct 30 08:00:05.946502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.946480    6738 scope.go:122] "RemoveContainer" containerID="6bfea46be9e3de2f7971d0e283811bef913a6f470666eb946df465a8a5704678"
Oct 30 08:00:05.962440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.962415    6738 scope.go:122] "RemoveContainer" containerID="74668cb0167ab28946388d5681abd874d539846a07743b94994bd7b95bd45540"
Oct 30 08:00:05.977790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.977768    6738 scope.go:122] "RemoveContainer" containerID="4d58438a9141593cc90963546a97f9e3c296d313de1ea34a0fd75eca9561f6a0"
Oct 30 08:00:05.992559 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:05.992533    6738 scope.go:122] "RemoveContainer" containerID="491c743e142b705f4422bc5811b96cc169d28adfa1b00bd70621b6c951b8754b"
Oct 30 08:00:06.007817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.007795    6738 scope.go:122] "RemoveContainer" containerID="e8f67b42ae8d0038ca2650a9fddbd06bedcae160f7817337e52ba696e2ff3571"
Oct 30 08:00:06.903997 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.903966    6738 scope.go:122] "RemoveContainer" containerID="2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"
Oct 30 08:00:06.904484 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.904371    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm_odh-ai-gateway-infra(51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"
Oct 30 08:00:06.905825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.905800    6738 scope.go:122] "RemoveContainer" containerID="f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"
Oct 30 08:00:06.906155 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.906130    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl_odh-ai-gateway-infra(c5232edb-fed4-493e-96f9-077bba327cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" podUID="c5232edb-fed4-493e-96f9-077bba327cca"
Oct 30 08:00:06.907480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.907455    6738 scope.go:122] "RemoveContainer" containerID="94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"
Oct 30 08:00:06.907810 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.907789    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl_odh-ai-gateway-infra(f8972292-ee0a-4d7a-be41-fb9d32dd4043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043"
Oct 30 08:00:06.909045 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.909027    6738 scope.go:122] "RemoveContainer" containerID="c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"
Oct 30 08:00:06.909374 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.909352    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5_odh-ai-gateway-infra(199c86b9-53fc-4184-8007-8639510fe71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" podUID="199c86b9-53fc-4184-8007-8639510fe71a"
Oct 30 08:00:06.910638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.910619    6738 scope.go:122] "RemoveContainer" containerID="4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"
Oct 30 08:00:06.910905 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.910885    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-bwz8n_odh-ai-gateway-infra(295bf5fa-d510-4779-84c5-fcbb3006bd62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62"
Oct 30 08:00:06.912130 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.912113    6738 scope.go:122] "RemoveContainer" containerID="6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"
Oct 30 08:00:06.912407 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.912388    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc_odh-ai-gateway-infra(920e7877-1932-4a3a-a845-b61357d5e804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" podUID="920e7877-1932-4a3a-a845-b61357d5e804"
Oct 30 08:00:06.913698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.913679    6738 scope.go:122] "RemoveContainer" containerID="9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"
Oct 30 08:00:06.913966 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.913943    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j_odh-ai-gateway-infra(0d904f9f-9c80-4d8d-a0be-13c1b179afe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1"
Oct 30 08:00:06.915227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.915209    6738 scope.go:122] "RemoveContainer" containerID="b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"
Oct 30 08:00:06.915505 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.915480    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w_odh-ai-gateway-infra(61865956-c60b-471e-a066-33cbee6cd28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" podUID="61865956-c60b-471e-a066-33cbee6cd28c"
Oct 30 08:00:06.916841 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.916823    6738 scope.go:122] "RemoveContainer" containerID="c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"
Oct 30 08:00:06.917049 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.917032    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd_odh-ai-gateway-infra(57826627-652f-4b1b-a66a-84d6075e13a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" podUID="57826627-652f-4b1b-a66a-84d6075e13a4"
Oct 30 08:00:06.918421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:06.918406    6738 scope.go:122] "RemoveContainer" containerID="2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"
Oct 30 08:00:06.918607 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:06.918592    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd_odh-ai-gateway-infra(a48710cf-4d3c-4d1c-a0dd-893da16b8bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6"
Oct 30 08:00:07.520567 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:07.520527    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:00:07.627279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:07.627242    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:09.435346 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:09.435279    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:09.856538 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:09.856454    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:00:11.292853 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:11.292816    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:13.162712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:13.162674    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:15.056120 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:15.056083    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:15.682897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:15.682866    6738 scope.go:122] "RemoveContainer" containerID="94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"
Oct 30 08:00:16.683568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:16.683531    6738 scope.go:122] "RemoveContainer" containerID="9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"
Oct 30 08:00:16.684046 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:16.683672    6738 scope.go:122] "RemoveContainer" containerID="b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"
Oct 30 08:00:16.934950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:16.934905    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:17.020785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.020752    6738 generic.go:356] "Generic (PLEG): container finished" podID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerID="4567151fdc8eed06a691627fb228fa37165110640d4c7a41a17c0611b9f6d567" exitCode=60
Oct 30 08:00:17.021012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.020825    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerDied","Data":"4567151fdc8eed06a691627fb228fa37165110640d4c7a41a17c0611b9f6d567"}
Oct 30 08:00:17.021012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.020885    6738 scope.go:122] "RemoveContainer" containerID="94af7d56b3bfd55a401e90e6758db98b2702d787dacd02e70e35cfa191b6749a"
Oct 30 08:00:17.021382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.021362    6738 scope.go:122] "RemoveContainer" containerID="4567151fdc8eed06a691627fb228fa37165110640d4c7a41a17c0611b9f6d567"
Oct 30 08:00:17.021792 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:17.021770    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl_odh-ai-gateway-infra(f8972292-ee0a-4d7a-be41-fb9d32dd4043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043"
Oct 30 08:00:17.683350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.683318    6738 scope.go:122] "RemoveContainer" containerID="4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"
Oct 30 08:00:17.683574 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.683549    6738 scope.go:122] "RemoveContainer" containerID="5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80"
Oct 30 08:00:17.684259 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.683845    6738 scope.go:122] "RemoveContainer" containerID="c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d"
Oct 30 08:00:17.684259 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.683941    6738 scope.go:122] "RemoveContainer" containerID="af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211"
Oct 30 08:00:17.684259 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.684029    6738 scope.go:122] "RemoveContainer" containerID="2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"
Oct 30 08:00:17.684259 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:17.684161    6738 scope.go:122] "RemoveContainer" containerID="2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"
Oct 30 08:00:18.034266 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.034233    6738 generic.go:356] "Generic (PLEG): container finished" podID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerID="901ad731eb80caec709a15cc52707c09f9b404c7b1adbaba18135aeeb9f3905c" exitCode=60
Oct 30 08:00:18.034491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.034325    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerDied","Data":"901ad731eb80caec709a15cc52707c09f9b404c7b1adbaba18135aeeb9f3905c"}
Oct 30 08:00:18.034491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.034390    6738 scope.go:122] "RemoveContainer" containerID="9a7a53e675cc170f624ed3d0176e0fedc02469f62ccc2c30cb283df4ec6f133c"
Oct 30 08:00:18.034866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.034830    6738 scope.go:122] "RemoveContainer" containerID="901ad731eb80caec709a15cc52707c09f9b404c7b1adbaba18135aeeb9f3905c"
Oct 30 08:00:18.035315 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:18.035254    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j_odh-ai-gateway-infra(0d904f9f-9c80-4d8d-a0be-13c1b179afe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1"
Oct 30 08:00:18.036888 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.036868    6738 generic.go:356] "Generic (PLEG): container finished" podID="61865956-c60b-471e-a066-33cbee6cd28c" containerID="c38c9e6d60527a3c050a4ab2f1e05b0bfd66574251fb6c167d0f730601877c8d" exitCode=60
Oct 30 08:00:18.036990 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.036941    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerDied","Data":"c38c9e6d60527a3c050a4ab2f1e05b0bfd66574251fb6c167d0f730601877c8d"}
Oct 30 08:00:18.037924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.037906    6738 scope.go:122] "RemoveContainer" containerID="c38c9e6d60527a3c050a4ab2f1e05b0bfd66574251fb6c167d0f730601877c8d"
Oct 30 08:00:18.038265 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:18.038247    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w_odh-ai-gateway-infra(61865956-c60b-471e-a066-33cbee6cd28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" podUID="61865956-c60b-471e-a066-33cbee6cd28c"
Oct 30 08:00:18.051923 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.051904    6738 scope.go:122] "RemoveContainer" containerID="b983ee6b29702879cddec0bb6a439a5ace2897404d02a64b666c05b3f4f03f11"
Oct 30 08:00:18.090861 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.090830    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:18.391064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.391036    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:18.547350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.547224    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\" (UniqueName: \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\") pod \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") "
Oct 30 08:00:18.547590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.547566    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\") pod \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\" (UID: \"f8972292-ee0a-4d7a-be41-fb9d32dd4043\") "
Oct 30 08:00:18.548120 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.548094    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca" pod "f8972292-ee0a-4d7a-be41-fb9d32dd4043" (UID: "f8972292-ee0a-4d7a-be41-fb9d32dd4043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.549816 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.549790    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc" pod "f8972292-ee0a-4d7a-be41-fb9d32dd4043" (UID: "f8972292-ee0a-4d7a-be41-fb9d32dd4043"). InnerVolumeSpecName "kube-api-access-jgsdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.649822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.649777    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8972292-ee0a-4d7a-be41-fb9d32dd4043-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.649822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.649822    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsdc\" (UniqueName: \"kubernetes.io/projected/f8972292-ee0a-4d7a-be41-fb9d32dd4043-kube-api-access-jgsdc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.682964 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.682937    6738 scope.go:122] "RemoveContainer" containerID="c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"
Oct 30 08:00:18.683117 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.683051    6738 scope.go:122] "RemoveContainer" containerID="f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"
Oct 30 08:00:18.683159 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.683127    6738 scope.go:122] "RemoveContainer" containerID="6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"
Oct 30 08:00:18.986818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:18.986786    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:19.075922 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.075885    6738 generic.go:356] "Generic (PLEG): container finished" podID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerID="cfdf3080f39473e5c8239b6f9bce682b85065b60304b006d81f541aac29180bc" exitCode=60
Oct 30 08:00:19.076090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.075966    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerDied","Data":"cfdf3080f39473e5c8239b6f9bce682b85065b60304b006d81f541aac29180bc"}
Oct 30 08:00:19.076090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.076066    6738 scope.go:122] "RemoveContainer" containerID="5d9c7b95d6f91790f46809ffe8b5c7848887f84b29e8b89054b6bc530a3e0c80"
Oct 30 08:00:19.076934 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.076811    6738 scope.go:122] "RemoveContainer" containerID="cfdf3080f39473e5c8239b6f9bce682b85065b60304b006d81f541aac29180bc"
Oct 30 08:00:19.077640 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:19.077264    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf_odh-ai-gateway-infra(ecd7ba37-da03-4528-a350-c4973aa9f9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4"
Oct 30 08:00:19.081577 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.080398    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerStarted","Data":"69d46ef1c3435528aef2486bbd918a9fdbf11ce33aa62567b678726bbe21a67f"}
Oct 30 08:00:19.083676 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.083650    6738 generic.go:356] "Generic (PLEG): container finished" podID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerID="01a4883a5e512a43e8c643d0ba7a75f8d7b3a3b09845f8fd90d6822b5431b280" exitCode=60
Oct 30 08:00:19.083786 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.083721    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" event={"ID":"6d5b65bd-7e5e-4d28-988a-dd2035b41572","Type":"ContainerDied","Data":"01a4883a5e512a43e8c643d0ba7a75f8d7b3a3b09845f8fd90d6822b5431b280"}
Oct 30 08:00:19.084112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.084095    6738 scope.go:122] "RemoveContainer" containerID="01a4883a5e512a43e8c643d0ba7a75f8d7b3a3b09845f8fd90d6822b5431b280"
Oct 30 08:00:19.084510 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:19.084483    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4_odh-ai-gateway-infra(6d5b65bd-7e5e-4d28-988a-dd2035b41572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572"
Oct 30 08:00:19.086694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.086672    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerStarted","Data":"270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048"}
Oct 30 08:00:19.087082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.087063    6738 scope.go:122] "RemoveContainer" containerID="270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048"
Oct 30 08:00:19.087467 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:19.087442    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm_odh-ai-gateway-infra(51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"
Oct 30 08:00:19.088269 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.088251    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"
Oct 30 08:00:19.088565 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.088263    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl" event={"ID":"f8972292-ee0a-4d7a-be41-fb9d32dd4043","Type":"ContainerDied","Data":"cd2db2fec7aef880cf3cbe8149ca1bd534e3641fd0f438e184615da54e784a7c"}
Oct 30 08:00:19.090356 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.090329    6738 generic.go:356] "Generic (PLEG): container finished" podID="147e81ec-5cb1-4b49-a878-e5f170457614" containerID="9d7d6366c30b99c93e45264422e7f36a18b0146a9fdcc5239e7a45a2eae3251b" exitCode=60
Oct 30 08:00:19.090455 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.090407    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" event={"ID":"147e81ec-5cb1-4b49-a878-e5f170457614","Type":"ContainerDied","Data":"9d7d6366c30b99c93e45264422e7f36a18b0146a9fdcc5239e7a45a2eae3251b"}
Oct 30 08:00:19.090789 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.090772    6738 scope.go:122] "RemoveContainer" containerID="9d7d6366c30b99c93e45264422e7f36a18b0146a9fdcc5239e7a45a2eae3251b"
Oct 30 08:00:19.091046 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:19.091029    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps_odh-ai-gateway-infra(147e81ec-5cb1-4b49-a878-e5f170457614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" podUID="147e81ec-5cb1-4b49-a878-e5f170457614"
Oct 30 08:00:19.092522 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.092500    6738 generic.go:356] "Generic (PLEG): container finished" podID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerID="44f3dceb08c0a357627d6fdde7d3ca04e09d7b4bb5364e46fa0f4e362fdab9a6" exitCode=60
Oct 30 08:00:19.092606 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.092570    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerDied","Data":"44f3dceb08c0a357627d6fdde7d3ca04e09d7b4bb5364e46fa0f4e362fdab9a6"}
Oct 30 08:00:19.093003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.092982    6738 scope.go:122] "RemoveContainer" containerID="44f3dceb08c0a357627d6fdde7d3ca04e09d7b4bb5364e46fa0f4e362fdab9a6"
Oct 30 08:00:19.093363 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:19.093274    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-bwz8n_odh-ai-gateway-infra(295bf5fa-d510-4779-84c5-fcbb3006bd62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62"
Oct 30 08:00:19.108939 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.108899    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:19.153877 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.153846    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:19.228548 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.227882    6738 scope.go:122] "RemoveContainer" containerID="af46722b006d1b3e4fa6831c4e136ca8f7a0c6709d2c9c3ced8b95cc74166211"
Oct 30 08:00:19.269502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.269477    6738 scope.go:122] "RemoveContainer" containerID="4567151fdc8eed06a691627fb228fa37165110640d4c7a41a17c0611b9f6d567"
Oct 30 08:00:19.288740 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.288434    6738 scope.go:122] "RemoveContainer" containerID="c94b53916143cb11b6781f2c4a1435f4c7a22f5f2c95e142df146d370c59157d"
Oct 30 08:00:19.298943 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.298911    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:19.313122 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.313087    6738 scope.go:122] "RemoveContainer" containerID="4cfb0b20f0f0737b276f854528f62c1082227ecd857ebc8668c1c46d5ed1f07e"
Oct 30 08:00:19.381188 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.381162    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:19.463400 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.463282    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\" (UniqueName: \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\") pod \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") "
Oct 30 08:00:19.463550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.463489    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\") pod \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\" (UID: \"0d904f9f-9c80-4d8d-a0be-13c1b179afe1\") "
Oct 30 08:00:19.463883 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.463857    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca" pod "0d904f9f-9c80-4d8d-a0be-13c1b179afe1" (UID: "0d904f9f-9c80-4d8d-a0be-13c1b179afe1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:19.464182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.464157    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.465577 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.465555    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv" pod "0d904f9f-9c80-4d8d-a0be-13c1b179afe1" (UID: "0d904f9f-9c80-4d8d-a0be-13c1b179afe1"). InnerVolumeSpecName "kube-api-access-kgmdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:19.486769 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.486735    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:19.493285 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.493240    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe2ed-29889120-mcjnl"]
Oct 30 08:00:19.564944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.564906    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\" (UniqueName: \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\") pod \"61865956-c60b-471e-a066-33cbee6cd28c\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") "
Oct 30 08:00:19.565223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.565183    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\") pod \"61865956-c60b-471e-a066-33cbee6cd28c\" (UID: \"61865956-c60b-471e-a066-33cbee6cd28c\") "
Oct 30 08:00:19.565599 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.565570    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca" pod "61865956-c60b-471e-a066-33cbee6cd28c" (UID: "61865956-c60b-471e-a066-33cbee6cd28c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:19.566047 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.566020    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgmdv\" (UniqueName: \"kubernetes.io/projected/0d904f9f-9c80-4d8d-a0be-13c1b179afe1-kube-api-access-kgmdv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.566204 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.566051    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61865956-c60b-471e-a066-33cbee6cd28c-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.567388 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.567358    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9" pod "61865956-c60b-471e-a066-33cbee6cd28c" (UID: "61865956-c60b-471e-a066-33cbee6cd28c"). InnerVolumeSpecName "kube-api-access-8mlm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:19.667343 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.667286    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mlm9\" (UniqueName: \"kubernetes.io/projected/61865956-c60b-471e-a066-33cbee6cd28c-kube-api-access-8mlm9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.690476 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.690442    6738 scope.go:122] "RemoveContainer" containerID="c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"
Oct 30 08:00:19.695562 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.695534    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" path="/var/lib/kubelet/pods/f8972292-ee0a-4d7a-be41-fb9d32dd4043/volumes"
Oct 30 08:00:19.856006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:19.855968    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:00:20.108732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.108699    6738 generic.go:356] "Generic (PLEG): container finished" podID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerID="69d46ef1c3435528aef2486bbd918a9fdbf11ce33aa62567b678726bbe21a67f" exitCode=60
Oct 30 08:00:20.109159 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.108779    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerDied","Data":"69d46ef1c3435528aef2486bbd918a9fdbf11ce33aa62567b678726bbe21a67f"}
Oct 30 08:00:20.109159 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.108851    6738 scope.go:122] "RemoveContainer" containerID="2301e826f72a1f2b2c70e507dd3513411c630d971ff7b5b69ab19a9406650e10"
Oct 30 08:00:20.109328 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.109283    6738 scope.go:122] "RemoveContainer" containerID="69d46ef1c3435528aef2486bbd918a9fdbf11ce33aa62567b678726bbe21a67f"
Oct 30 08:00:20.109683 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:20.109664    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd_odh-ai-gateway-infra(a48710cf-4d3c-4d1c-a0dd-893da16b8bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6"
Oct 30 08:00:20.114032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.114005    6738 generic.go:356] "Generic (PLEG): container finished" podID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerID="270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048" exitCode=60
Oct 30 08:00:20.114163 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.114089    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerDied","Data":"270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048"}
Oct 30 08:00:20.114592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.114573    6738 scope.go:122] "RemoveContainer" containerID="270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048"
Oct 30 08:00:20.114962 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:20.114942    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm_odh-ai-gateway-infra(51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"
Oct 30 08:00:20.116395 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.116373    6738 generic.go:356] "Generic (PLEG): container finished" podID="c5232edb-fed4-493e-96f9-077bba327cca" containerID="a6474d4c111eebeac46b641fb84d499172309c9b707402e014e2401bdd728751" exitCode=60
Oct 30 08:00:20.116506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.116442    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerDied","Data":"a6474d4c111eebeac46b641fb84d499172309c9b707402e014e2401bdd728751"}
Oct 30 08:00:20.116886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.116829    6738 scope.go:122] "RemoveContainer" containerID="a6474d4c111eebeac46b641fb84d499172309c9b707402e014e2401bdd728751"
Oct 30 08:00:20.117207 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:20.117181    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl_odh-ai-gateway-infra(c5232edb-fed4-493e-96f9-077bba327cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" podUID="c5232edb-fed4-493e-96f9-077bba327cca"
Oct 30 08:00:20.119817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.119796    6738 generic.go:356] "Generic (PLEG): container finished" podID="199c86b9-53fc-4184-8007-8639510fe71a" containerID="867336d3f3d97e3cbb89496bf177d249177023f4a800270644ceb33583416287" exitCode=60
Oct 30 08:00:20.119921 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.119860    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerDied","Data":"867336d3f3d97e3cbb89496bf177d249177023f4a800270644ceb33583416287"}
Oct 30 08:00:20.120176 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.120159    6738 scope.go:122] "RemoveContainer" containerID="867336d3f3d97e3cbb89496bf177d249177023f4a800270644ceb33583416287"
Oct 30 08:00:20.120513 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:20.120485    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5_odh-ai-gateway-infra(199c86b9-53fc-4184-8007-8639510fe71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" podUID="199c86b9-53fc-4184-8007-8639510fe71a"
Oct 30 08:00:20.126168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.126147    6738 generic.go:356] "Generic (PLEG): container finished" podID="920e7877-1932-4a3a-a845-b61357d5e804" containerID="b8c6f9593156be0d0299eb32ef40e676e0db9e70b1211bf12da93762901882ed" exitCode=60
Oct 30 08:00:20.126278 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.126171    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerDied","Data":"b8c6f9593156be0d0299eb32ef40e676e0db9e70b1211bf12da93762901882ed"}
Oct 30 08:00:20.126568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.126549    6738 scope.go:122] "RemoveContainer" containerID="b8c6f9593156be0d0299eb32ef40e676e0db9e70b1211bf12da93762901882ed"
Oct 30 08:00:20.126902 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:20.126879    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc_odh-ai-gateway-infra(920e7877-1932-4a3a-a845-b61357d5e804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" podUID="920e7877-1932-4a3a-a845-b61357d5e804"
Oct 30 08:00:20.127711 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.127695    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"
Oct 30 08:00:20.127711 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.127701    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j" event={"ID":"0d904f9f-9c80-4d8d-a0be-13c1b179afe1","Type":"ContainerDied","Data":"86f1f338fa19cc590e67f59695a830af213236accde2fe293386adc610a5aba0"}
Oct 30 08:00:20.129119 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.129103    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"
Oct 30 08:00:20.129217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.129106    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w" event={"ID":"61865956-c60b-471e-a066-33cbee6cd28c","Type":"ContainerDied","Data":"a6242840df71b3e8a24ddb1e8bbd11c4eb970c7ce4e78084fb22b97fc2a8b434"}
Oct 30 08:00:20.146708 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.146675    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:20.191644 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.191597    6738 scope.go:122] "RemoveContainer" containerID="2636a83a8d40071ae1d061f007d8aa83834ef05b41b8faa30633009760fb1166"
Oct 30 08:00:20.209354 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.202758    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:20.229813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.229199    6738 scope.go:122] "RemoveContainer" containerID="f2da3b2882a24e34b992db59565b8895728c95cd74f425f8d1c7ef7debc26aeb"
Oct 30 08:00:20.267082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.266251    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:20.296701 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.296668    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:20.312760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.310499    6738 scope.go:122] "RemoveContainer" containerID="c6a26d9991063ea26c3119fd9af8205357c87135864185d78e23ece33414da92"
Oct 30 08:00:20.352456 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.351585    6738 scope.go:122] "RemoveContainer" containerID="6fc3e5c5c41226b96bf2fe8c8a7c937bcdfed2452ea53334329bbe6eb0f71659"
Oct 30 08:00:20.374106 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.374035    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:20.386322 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.382158    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:20.411590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.408673    6738 scope.go:122] "RemoveContainer" containerID="901ad731eb80caec709a15cc52707c09f9b404c7b1adbaba18135aeeb9f3905c"
Oct 30 08:00:20.425667 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.425252    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:20.463563 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.459467    6738 scope.go:122] "RemoveContainer" containerID="c38c9e6d60527a3c050a4ab2f1e05b0bfd66574251fb6c167d0f730601877c8d"
Oct 30 08:00:20.488788 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.488759    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\") pod \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") "
Oct 30 08:00:20.489167 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.489142    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca" pod "ecd7ba37-da03-4528-a350-c4973aa9f9f4" (UID: "ecd7ba37-da03-4528-a350-c4973aa9f9f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.489332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.489155    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\" (UniqueName: \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\") pod \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\" (UID: \"ecd7ba37-da03-4528-a350-c4973aa9f9f4\") "
Oct 30 08:00:20.490008 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.489980    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd7ba37-da03-4528-a350-c4973aa9f9f4-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.509655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.509618    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d" pod "ecd7ba37-da03-4528-a350-c4973aa9f9f4" (UID: "ecd7ba37-da03-4528-a350-c4973aa9f9f4"). InnerVolumeSpecName "kube-api-access-jbk7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.546064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.546027    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:20.557362 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.557333    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:20.583072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.583045    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:20.591174 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.591140    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbk7d\" (UniqueName: \"kubernetes.io/projected/ecd7ba37-da03-4528-a350-c4973aa9f9f4-kube-api-access-jbk7d\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.673334 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.673286    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:20.677872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.677847    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bec5c-29889120-mcd7j"]
Oct 30 08:00:20.691599 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691566    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\") pod \"147e81ec-5cb1-4b49-a878-e5f170457614\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") "
Oct 30 08:00:20.691749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691706    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\" (UniqueName: \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\") pod \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") "
Oct 30 08:00:20.691749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691733    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\") pod \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\" (UID: \"6d5b65bd-7e5e-4d28-988a-dd2035b41572\") "
Oct 30 08:00:20.691833 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691773    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\" (UniqueName: \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\") pod \"295bf5fa-d510-4779-84c5-fcbb3006bd62\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") "
Oct 30 08:00:20.691904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691889    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\") pod \"295bf5fa-d510-4779-84c5-fcbb3006bd62\" (UID: \"295bf5fa-d510-4779-84c5-fcbb3006bd62\") "
Oct 30 08:00:20.691983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691953    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\" (UniqueName: \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\") pod \"147e81ec-5cb1-4b49-a878-e5f170457614\" (UID: \"147e81ec-5cb1-4b49-a878-e5f170457614\") "
Oct 30 08:00:20.692051 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.691998    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca" pod "147e81ec-5cb1-4b49-a878-e5f170457614" (UID: "147e81ec-5cb1-4b49-a878-e5f170457614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.692217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.692116    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca" pod "6d5b65bd-7e5e-4d28-988a-dd2035b41572" (UID: "6d5b65bd-7e5e-4d28-988a-dd2035b41572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.692343 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.692285    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca" pod "295bf5fa-d510-4779-84c5-fcbb3006bd62" (UID: "295bf5fa-d510-4779-84c5-fcbb3006bd62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.692694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.692676    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295bf5fa-d510-4779-84c5-fcbb3006bd62-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.692777 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.692699    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147e81ec-5cb1-4b49-a878-e5f170457614-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.692777 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.692715    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d5b65bd-7e5e-4d28-988a-dd2035b41572-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.694009 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.693990    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k" pod "147e81ec-5cb1-4b49-a878-e5f170457614" (UID: "147e81ec-5cb1-4b49-a878-e5f170457614"). InnerVolumeSpecName "kube-api-access-rvs4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.694403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.694383    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4" pod "6d5b65bd-7e5e-4d28-988a-dd2035b41572" (UID: "6d5b65bd-7e5e-4d28-988a-dd2035b41572"). InnerVolumeSpecName "kube-api-access-jqvz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.694458 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.694413    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p" pod "295bf5fa-d510-4779-84c5-fcbb3006bd62" (UID: "295bf5fa-d510-4779-84c5-fcbb3006bd62"). InnerVolumeSpecName "kube-api-access-5fd4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.709882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.709849    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:20.714223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.714194    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889120-nfk9w"]
Oct 30 08:00:20.794364 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.794225    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvz4\" (UniqueName: \"kubernetes.io/projected/6d5b65bd-7e5e-4d28-988a-dd2035b41572-kube-api-access-jqvz4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.794364 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.794260    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fd4p\" (UniqueName: \"kubernetes.io/projected/295bf5fa-d510-4779-84c5-fcbb3006bd62-kube-api-access-5fd4p\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.794364 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:20.794276    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvs4k\" (UniqueName: \"kubernetes.io/projected/147e81ec-5cb1-4b49-a878-e5f170457614-kube-api-access-rvs4k\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.143048 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.143015    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"
Oct 30 08:00:21.143550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.143045    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps" event={"ID":"147e81ec-5cb1-4b49-a878-e5f170457614","Type":"ContainerDied","Data":"d177647a92cf8db936a88c7d13ccead468843520039a7c38778ab59903ae4255"}
Oct 30 08:00:21.143550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.143119    6738 scope.go:122] "RemoveContainer" containerID="9d7d6366c30b99c93e45264422e7f36a18b0146a9fdcc5239e7a45a2eae3251b"
Oct 30 08:00:21.144818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.144732    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"
Oct 30 08:00:21.144818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.144775    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n" event={"ID":"295bf5fa-d510-4779-84c5-fcbb3006bd62","Type":"ContainerDied","Data":"e350444d6067167c7c9a6b93d50549ee8423d7c76196b916eacbe081c3401385"}
Oct 30 08:00:21.152131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.152098    6738 generic.go:356] "Generic (PLEG): container finished" podID="57826627-652f-4b1b-a66a-84d6075e13a4" containerID="189ecbd04347eff28188019c4974e0ab7beb65bb7047a7c19474bb223c514836" exitCode=60
Oct 30 08:00:21.152289 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.152185    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerDied","Data":"189ecbd04347eff28188019c4974e0ab7beb65bb7047a7c19474bb223c514836"}
Oct 30 08:00:21.152725 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.152701    6738 scope.go:122] "RemoveContainer" containerID="189ecbd04347eff28188019c4974e0ab7beb65bb7047a7c19474bb223c514836"
Oct 30 08:00:21.153112 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:21.153088    6738 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd_odh-ai-gateway-infra(57826627-652f-4b1b-a66a-84d6075e13a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" podUID="57826627-652f-4b1b-a66a-84d6075e13a4"
Oct 30 08:00:21.153785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.153759    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"
Oct 30 08:00:21.153889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.153806    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf" event={"ID":"ecd7ba37-da03-4528-a350-c4973aa9f9f4","Type":"ContainerDied","Data":"63741188ca8dd98a7983c02776f134b6c03904fba833f2a5200fad7924f712e3"}
Oct 30 08:00:21.158181 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.158131    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4" event={"ID":"6d5b65bd-7e5e-4d28-988a-dd2035b41572","Type":"ContainerDied","Data":"e47da4a934c4e39bed2ae4a58d15b1e5053eea581c94a38034be95a088ca5384"}
Oct 30 08:00:21.158397 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.158374    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"
Oct 30 08:00:21.159498 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.159480    6738 scope.go:122] "RemoveContainer" containerID="44f3dceb08c0a357627d6fdde7d3ca04e09d7b4bb5364e46fa0f4e362fdab9a6"
Oct 30 08:00:21.179884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.179856    6738 scope.go:122] "RemoveContainer" containerID="c87a9d80fbce2ef6a66af80e81ce12cf5a83d32242b6cc5fffe035a4b1405824"
Oct 30 08:00:21.199596 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.199569    6738 scope.go:122] "RemoveContainer" containerID="cfdf3080f39473e5c8239b6f9bce682b85065b60304b006d81f541aac29180bc"
Oct 30 08:00:21.215556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.215527    6738 scope.go:122] "RemoveContainer" containerID="01a4883a5e512a43e8c643d0ba7a75f8d7b3a3b09845f8fd90d6822b5431b280"
Oct 30 08:00:21.219379 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.219347    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:21.233020 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.232982    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a475d6-29889120-k46ps"]
Oct 30 08:00:21.249080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.248758    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:21.330546 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.330208    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:21.403084 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.402992    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:21.417477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.417208    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:21.432622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.431645    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:21.432622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.432087    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\") pod \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") "
Oct 30 08:00:21.432622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.432244    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\" (UniqueName: \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\") pod \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\" (UID: \"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6\") "
Oct 30 08:00:21.444866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.443846    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca" pod "a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" (UID: "a48710cf-4d3c-4d1c-a0dd-893da16b8bf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.444866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.443992    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j" pod "a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" (UID: "a48710cf-4d3c-4d1c-a0dd-893da16b8bf6"). InnerVolumeSpecName "kube-api-access-jpw8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.445841 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.445207    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-bwz8n"]
Oct 30 08:00:21.486360 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.486324    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:21.486597 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.486579    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:21.537713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.537662    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.537933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.537917    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpw8j\" (UniqueName: \"kubernetes.io/projected/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6-kube-api-access-jpw8j\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.563760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.563732    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:21.586510 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.586388    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:21.593252 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.593217    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1f34aa-29889120-lwgr4"]
Oct 30 08:00:21.640759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.639135    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\") pod \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") "
Oct 30 08:00:21.640759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.639269    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\" (UniqueName: \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\") pod \"199c86b9-53fc-4184-8007-8639510fe71a\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") "
Oct 30 08:00:21.640759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.639351    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\") pod \"199c86b9-53fc-4184-8007-8639510fe71a\" (UID: \"199c86b9-53fc-4184-8007-8639510fe71a\") "
Oct 30 08:00:21.640759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.639461    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\" (UniqueName: \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\") pod \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\" (UID: \"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32\") "
Oct 30 08:00:21.644085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.643322    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca" pod "199c86b9-53fc-4184-8007-8639510fe71a" (UID: "199c86b9-53fc-4184-8007-8639510fe71a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.644085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.643698    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca" pod "51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" (UID: "51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.648771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.648726    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5" pod "51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" (UID: "51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32"). InnerVolumeSpecName "kube-api-access-7rql5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.652120 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.652092    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f" pod "199c86b9-53fc-4184-8007-8639510fe71a" (UID: "199c86b9-53fc-4184-8007-8639510fe71a"). InnerVolumeSpecName "kube-api-access-b488f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.656595 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.656527    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:21.677123 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.677081    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:21.681210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.681180    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8d0c2f-29889120-2dswf"]
Oct 30 08:00:21.697444 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.697410    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" path="/var/lib/kubelet/pods/0d904f9f-9c80-4d8d-a0be-13c1b179afe1/volumes"
Oct 30 08:00:21.698098 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.698082    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" path="/var/lib/kubelet/pods/147e81ec-5cb1-4b49-a878-e5f170457614/volumes"
Oct 30 08:00:21.698688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.698672    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" path="/var/lib/kubelet/pods/295bf5fa-d510-4779-84c5-fcbb3006bd62/volumes"
Oct 30 08:00:21.699717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.699699    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61865956-c60b-471e-a066-33cbee6cd28c" path="/var/lib/kubelet/pods/61865956-c60b-471e-a066-33cbee6cd28c/volumes"
Oct 30 08:00:21.700265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.700250    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" path="/var/lib/kubelet/pods/6d5b65bd-7e5e-4d28-988a-dd2035b41572/volumes"
Oct 30 08:00:21.700842 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.700825    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" path="/var/lib/kubelet/pods/ecd7ba37-da03-4528-a350-c4973aa9f9f4/volumes"
Oct 30 08:00:21.741604 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.741567    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\") pod \"920e7877-1932-4a3a-a845-b61357d5e804\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") "
Oct 30 08:00:21.741793 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.741719    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\" (UniqueName: \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\") pod \"920e7877-1932-4a3a-a845-b61357d5e804\" (UID: \"920e7877-1932-4a3a-a845-b61357d5e804\") "
Oct 30 08:00:21.741990 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.741963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca" pod "920e7877-1932-4a3a-a845-b61357d5e804" (UID: "920e7877-1932-4a3a-a845-b61357d5e804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.742321 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.742284    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rql5\" (UniqueName: \"kubernetes.io/projected/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-kube-api-access-7rql5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.742404 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.742325    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920e7877-1932-4a3a-a845-b61357d5e804-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.742404 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.742335    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.742404 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.742345    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b488f\" (UniqueName: \"kubernetes.io/projected/199c86b9-53fc-4184-8007-8639510fe71a-kube-api-access-b488f\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.742404 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.742353    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199c86b9-53fc-4184-8007-8639510fe71a-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.743957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.743933    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm" pod "920e7877-1932-4a3a-a845-b61357d5e804" (UID: "920e7877-1932-4a3a-a845-b61357d5e804"). InnerVolumeSpecName "kube-api-access-2qdnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.768827 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.768800    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:21.843328 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.843276    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\") pod \"c5232edb-fed4-493e-96f9-077bba327cca\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") "
Oct 30 08:00:21.843572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.843509    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\" (UniqueName: \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\") pod \"c5232edb-fed4-493e-96f9-077bba327cca\" (UID: \"c5232edb-fed4-493e-96f9-077bba327cca\") "
Oct 30 08:00:21.844515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.844487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca" pod "c5232edb-fed4-493e-96f9-077bba327cca" (UID: "c5232edb-fed4-493e-96f9-077bba327cca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.844593 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.844572    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdnm\" (UniqueName: \"kubernetes.io/projected/920e7877-1932-4a3a-a845-b61357d5e804-kube-api-access-2qdnm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.847141 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.847102    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9" pod "c5232edb-fed4-493e-96f9-077bba327cca" (UID: "c5232edb-fed4-493e-96f9-077bba327cca"). InnerVolumeSpecName "kube-api-access-vlpd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.946201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.946115    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5232edb-fed4-493e-96f9-077bba327cca-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.946201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:21.946148    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpd9\" (UniqueName: \"kubernetes.io/projected/c5232edb-fed4-493e-96f9-077bba327cca-kube-api-access-vlpd9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:22.180917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.180888    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"
Oct 30 08:00:22.180917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.180910    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd" event={"ID":"a48710cf-4d3c-4d1c-a0dd-893da16b8bf6","Type":"ContainerDied","Data":"fa6bc9b4ce49991703069b78cde614f3b638662a58cb96fad462cd2c15fc0ef4"}
Oct 30 08:00:22.181513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.180972    6738 scope.go:122] "RemoveContainer" containerID="69d46ef1c3435528aef2486bbd918a9fdbf11ce33aa62567b678726bbe21a67f"
Oct 30 08:00:22.183736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.183712    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"
Oct 30 08:00:22.183868 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.183731    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm" event={"ID":"51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32","Type":"ContainerDied","Data":"64165c97cb009c47c93e04f927efc3d4c006fc35f25ec27ce167fe6f8c7a8fbb"}
Oct 30 08:00:22.185438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.185408    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"
Oct 30 08:00:22.185570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.185410    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl" event={"ID":"c5232edb-fed4-493e-96f9-077bba327cca","Type":"ContainerDied","Data":"4eedf640e2eba51bcd38837a6461fd8deac3fceb7ec6ab487d29fccd719ea57d"}
Oct 30 08:00:22.186964 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.186938    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5" event={"ID":"199c86b9-53fc-4184-8007-8639510fe71a","Type":"ContainerDied","Data":"83fbe1dd8bf6b0d5dccfe210c51308c6cba8d9770faf5b2b69dba5a4f4d24615"}
Oct 30 08:00:22.187093 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.186945    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"
Oct 30 08:00:22.190920 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.190898    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"
Oct 30 08:00:22.191059 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.190899    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc" event={"ID":"920e7877-1932-4a3a-a845-b61357d5e804","Type":"ContainerDied","Data":"e15855becd5c2f6f1928618e871e6faefe32d17838c133e8fa838b8c15a4dd1a"}
Oct 30 08:00:22.193993 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.193974    6738 scope.go:122] "RemoveContainer" containerID="270172fa6dfc8b72cfe43b7d778056ec94eb4413fa7fd462b55030ca6b2e0048"
Oct 30 08:00:22.205824 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.205801    6738 scope.go:122] "RemoveContainer" containerID="a6474d4c111eebeac46b641fb84d499172309c9b707402e014e2401bdd728751"
Oct 30 08:00:22.216302 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.216266    6738 scope.go:122] "RemoveContainer" containerID="867336d3f3d97e3cbb89496bf177d249177023f4a800270644ceb33583416287"
Oct 30 08:00:22.230271 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.230241    6738 scope.go:122] "RemoveContainer" containerID="b8c6f9593156be0d0299eb32ef40e676e0db9e70b1211bf12da93762901882ed"
Oct 30 08:00:22.273892 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.273851    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:22.301954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.301917    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889120-b2qlm"]
Oct 30 08:00:22.355063 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.355020    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:22.377846 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.377802    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c1b5d7-29889120-n9vtc"]
Oct 30 08:00:22.412384 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.412352    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:22.418672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.418643    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889120-rxvfl"]
Oct 30 08:00:22.455356 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.455246    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:22.458767 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.458734    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-06e19a-29889120-wdfk5"]
Oct 30 08:00:22.488850 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.488821    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:22.501348 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.501313    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:22.508598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.508569    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f5ceca-29889120-xffzd"]
Oct 30 08:00:22.647759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.647733    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:22.757891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.757814    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\" (UniqueName: \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\") pod \"57826627-652f-4b1b-a66a-84d6075e13a4\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") "
Oct 30 08:00:22.758051 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.757926    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\") pod \"57826627-652f-4b1b-a66a-84d6075e13a4\" (UID: \"57826627-652f-4b1b-a66a-84d6075e13a4\") "
Oct 30 08:00:22.758338 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.758314    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca" pod "57826627-652f-4b1b-a66a-84d6075e13a4" (UID: "57826627-652f-4b1b-a66a-84d6075e13a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:22.758682 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.758663    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57826627-652f-4b1b-a66a-84d6075e13a4-maas-api-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:22.759869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.759844    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8" pod "57826627-652f-4b1b-a66a-84d6075e13a4" (UID: "57826627-652f-4b1b-a66a-84d6075e13a4"). InnerVolumeSpecName "kube-api-access-zjgt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:22.859596 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:22.859560    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgt8\" (UniqueName: \"kubernetes.io/projected/57826627-652f-4b1b-a66a-84d6075e13a4-kube-api-access-zjgt8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:23.202624 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.202592    6738 generic.go:356] "Generic (PLEG): container finished" podID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerID="40284472999dff81a91eb3d1f79269d0c16cc228f17c1abfb1978e0cb6a1d6ba" exitCode=0
Oct 30 08:00:23.203050 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.202668    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" event={"ID":"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd","Type":"ContainerDied","Data":"40284472999dff81a91eb3d1f79269d0c16cc228f17c1abfb1978e0cb6a1d6ba"}
Oct 30 08:00:23.203050 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.202716    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b" event={"ID":"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd","Type":"ContainerDied","Data":"12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243"}
Oct 30 08:00:23.203050 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.202740    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12b42aa546e00f281ac6aa7bc8ad0ce5a2f517a8e83adf48fc4fe413550dc243"
Oct 30 08:00:23.205399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.205375    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"
Oct 30 08:00:23.205539 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.205408    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd" event={"ID":"57826627-652f-4b1b-a66a-84d6075e13a4","Type":"ContainerDied","Data":"33f8ffcc3bd533bcba9f88196fbbf7e7dc5d1ad743dcc63da45d1d82ba943020"}
Oct 30 08:00:23.205539 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.205471    6738 scope.go:122] "RemoveContainer" containerID="189ecbd04347eff28188019c4974e0ab7beb65bb7047a7c19474bb223c514836"
Oct 30 08:00:23.222611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.222591    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 08:00:23.260707 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.260672    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:23.265918 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.265888    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\") pod \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") "
Oct 30 08:00:23.266083 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.266032    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\" (UniqueName: \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\") pod \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\" (UID: \"dfcba59f-2e73-455b-b7a2-416e1f7ad1bd\") "
Oct 30 08:00:23.266353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.266332    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume" pod "dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" (UID: "dfcba59f-2e73-455b-b7a2-416e1f7ad1bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:23.266681 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.266637    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:23.268434 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.268406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn" pod "dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" (UID: "dfcba59f-2e73-455b-b7a2-416e1f7ad1bd"). InnerVolumeSpecName "kube-api-access-q6jwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:23.282347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.275787    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1f700-29889120-qtvqd"]
Oct 30 08:00:23.323120 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.323022    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:23.370188 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.370149    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jwn\" (UniqueName: \"kubernetes.io/projected/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd-kube-api-access-q6jwn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:23.691256 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.691212    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199c86b9-53fc-4184-8007-8639510fe71a" path="/var/lib/kubelet/pods/199c86b9-53fc-4184-8007-8639510fe71a/volumes"
Oct 30 08:00:23.691819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.691796    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" path="/var/lib/kubelet/pods/51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32/volumes"
Oct 30 08:00:23.692331 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.692284    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" path="/var/lib/kubelet/pods/57826627-652f-4b1b-a66a-84d6075e13a4/volumes"
Oct 30 08:00:23.693112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.693090    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920e7877-1932-4a3a-a845-b61357d5e804" path="/var/lib/kubelet/pods/920e7877-1932-4a3a-a845-b61357d5e804/volumes"
Oct 30 08:00:23.693582 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.693562    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" path="/var/lib/kubelet/pods/a48710cf-4d3c-4d1c-a0dd-893da16b8bf6/volumes"
Oct 30 08:00:23.694011 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:23.693993    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5232edb-fed4-493e-96f9-077bba327cca" path="/var/lib/kubelet/pods/c5232edb-fed4-493e-96f9-077bba327cca/volumes"
Oct 30 08:00:24.219363 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:24.219333    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"
Oct 30 08:00:24.273383 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:24.273343    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 08:00:24.285823 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:24.285789    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1f34aa-786554f89d-vtd9b"]
Oct 30 08:00:25.163408 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:25.163372    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:25.691691 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:25.691650    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" path="/var/lib/kubelet/pods/dfcba59f-2e73-455b-b7a2-416e1f7ad1bd/volumes"
Oct 30 08:00:26.952126 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:26.952090    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:28.827692 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:28.827653    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:30.753764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:30.749579    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:30.988909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:30.988875    6738 scope.go:122] "RemoveContainer" containerID="e1fce70610107e3a3ad8bd3db86f2a7031d84f571ef68e22e44b668d6528b6de"
Oct 30 08:00:30.999651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:30.999627    6738 scope.go:122] "RemoveContainer" containerID="40284472999dff81a91eb3d1f79269d0c16cc228f17c1abfb1978e0cb6a1d6ba"
Oct 30 08:00:31.010236 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:31.010211    6738 scope.go:122] "RemoveContainer" containerID="c498f24aba63068bbb44faaa7300b00e68448fd04a73c5a54d15a61b8c206e21"
Oct 30 08:00:31.020478 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:31.020454    6738 scope.go:122] "RemoveContainer" containerID="a0cdae491c4de8ecad1319cac5c7e258023aad3f96dff5fe131f5718dfa943d6"
Oct 30 08:00:31.030308 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:31.030274    6738 scope.go:122] "RemoveContainer" containerID="91dbda671dfb7b373d186076529339e0c072611b6bfbe779052e6eb33fa0e367"
Oct 30 08:00:32.697226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:32.697194    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:34.558074 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:34.558035    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:35.501499 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:35.501461    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:35.622220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:35.622180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:36.032049 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:36.032006    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"]
Oct 30 08:00:36.043955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:36.043923    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0c763c-jchw4"]
Oct 30 08:00:36.313973 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:36.313885    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:37.521944 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:37.521905    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:00:37.691980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:37.691944    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30" path="/var/lib/kubelet/pods/6ef2cd1a-a06d-4b82-8c7b-e7b4de1ddd30/volumes"
Oct 30 08:00:38.128663 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:38.128627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:39.666906 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:39.666857    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:40.275554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.275520    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:40.415147 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.415107    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:40.525956 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.525865    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"]
Oct 30 08:00:40.532902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.532875    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1f34aa-9f4kt"]
Oct 30 08:00:40.673321 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.671007    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:40.741026 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.740983    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"]
Oct 30 08:00:40.741847 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741822    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.741847 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741846    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741869    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741877    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741887    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741895    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741910    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741917    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741927    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741934    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741944    6738 cpu_manager.go:414] "containerMap: removing container" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741952    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741965    6738 cpu_manager.go:414] "containerMap: removing container" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741973    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741987    6738 cpu_manager.go:414] "containerMap: removing container" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.741995    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742006    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742014    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742032    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.742041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742038    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742053    6738 cpu_manager.go:414] "containerMap: removing container" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742060    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742077    6738 cpu_manager.go:414] "containerMap: removing container" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742084    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742095    6738 cpu_manager.go:414] "containerMap: removing container" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742103    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742117    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742125    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742139    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742146    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742157    6738 cpu_manager.go:414] "containerMap: removing container" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742165    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742180    6738 cpu_manager.go:414] "containerMap: removing container" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742187    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742199    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742206    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742220    6738 cpu_manager.go:414] "containerMap: removing container" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742226    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742243    6738 cpu_manager.go:414] "containerMap: removing container" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742251    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742260    6738 cpu_manager.go:414] "containerMap: removing container" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742268    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742276    6738 cpu_manager.go:414] "containerMap: removing container" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742283    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742312    6738 cpu_manager.go:414] "containerMap: removing container" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742320    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742329    6738 cpu_manager.go:414] "containerMap: removing container" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742337    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742348    6738 cpu_manager.go:414] "containerMap: removing container" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742355    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742389    6738 cpu_manager.go:414] "containerMap: removing container" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742397    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742409    6738 cpu_manager.go:414] "containerMap: removing container" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742416    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742430    6738 cpu_manager.go:414] "containerMap: removing container" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742437    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742453    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742460    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742471    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742479    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742492    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742499    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742707    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742722    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742734    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742746    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742756    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742765    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742772    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="295bf5fa-d510-4779-84c5-fcbb3006bd62" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742784    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742792    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742804    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742811    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742822    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742831    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742839    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742851    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742862    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742873    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742884    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742891    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742901    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742911    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742920    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742931    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742941    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcba59f-2e73-455b-b7a2-416e1f7ad1bd" containerName="payload-processing"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742950    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742959    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742970    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742980    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.742989    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8972292-ee0a-4d7a-be41-fb9d32dd4043" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743000    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743008    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743018    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="199c86b9-53fc-4184-8007-8639510fe71a" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743027    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743037    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:40.743069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.743047    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="920e7877-1932-4a3a-a845-b61357d5e804" containerName="cleanup"
Oct 30 08:00:40.749502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.749478    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.762667 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.762637    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"]
Oct 30 08:00:40.764640 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.764609    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:00:40.764869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.764848    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:00:40.875443 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.875405    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcbp\" (UniqueName: \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.875678 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.875624    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.875763 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.875681    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.952904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.952867    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:40.976962 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.976926    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcbp\" (UniqueName: \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.977149 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.977057    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.977149 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.977090    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.977859 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.977834    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:40.979628 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:40.979600    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:41.001064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:41.001031    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcbp\" (UniqueName: \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\") pod \"maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:41.066215 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:41.066177    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:41.373210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:41.373179    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"]
Oct 30 08:00:41.373642 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:00:41.373616    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d7c427_c0a1_4e19_aaeb_5a255674871c.slice/crio-7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7 WatchSource:0}: Error finding container 7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7: Status 404 returned error can't find the container with id 7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7
Oct 30 08:00:41.693604 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:41.693523    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe114a42-050a-4625-a659-8d8202f81728" path="/var/lib/kubelet/pods/fe114a42-050a-4625-a659-8d8202f81728/volumes"
Oct 30 08:00:42.377407 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:42.377375    6738 generic.go:356] "Generic (PLEG): container finished" podID="60d7c427-c0a1-4e19-aaeb-5a255674871c" containerID="a7e0f0f9db05f387546f19578684147e3cb88aaf8b2c9e1a78e504b3ebb79dea" exitCode=0
Oct 30 08:00:42.377598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:42.377462    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z" event={"ID":"60d7c427-c0a1-4e19-aaeb-5a255674871c","Type":"ContainerDied","Data":"a7e0f0f9db05f387546f19578684147e3cb88aaf8b2c9e1a78e504b3ebb79dea"}
Oct 30 08:00:42.377598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:42.377516    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z" event={"ID":"60d7c427-c0a1-4e19-aaeb-5a255674871c","Type":"ContainerStarted","Data":"7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7"}
Oct 30 08:00:42.677141 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:42.677089    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:43.582743 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.582715    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:43.718513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.718427    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\" (UniqueName: \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\") pod \"60d7c427-c0a1-4e19-aaeb-5a255674871c\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") "
Oct 30 08:00:43.718778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.718609    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\") pod \"60d7c427-c0a1-4e19-aaeb-5a255674871c\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") "
Oct 30 08:00:43.718778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.718677    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\") pod \"60d7c427-c0a1-4e19-aaeb-5a255674871c\" (UID: \"60d7c427-c0a1-4e19-aaeb-5a255674871c\") "
Oct 30 08:00:43.719083 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.719058    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca" pod "60d7c427-c0a1-4e19-aaeb-5a255674871c" (UID: "60d7c427-c0a1-4e19-aaeb-5a255674871c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:43.719269 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.719253    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:43.720629 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.720598    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca" pod "60d7c427-c0a1-4e19-aaeb-5a255674871c" (UID: "60d7c427-c0a1-4e19-aaeb-5a255674871c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:43.720788 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.720689    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp" pod "60d7c427-c0a1-4e19-aaeb-5a255674871c" (UID: "60d7c427-c0a1-4e19-aaeb-5a255674871c"). InnerVolumeSpecName "kube-api-access-rpcbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:43.820890 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.820853    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d7c427-c0a1-4e19-aaeb-5a255674871c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:43.820890 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:43.820886    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcbp\" (UniqueName: \"kubernetes.io/projected/60d7c427-c0a1-4e19-aaeb-5a255674871c-kube-api-access-rpcbp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:44.397753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:44.397719    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"
Oct 30 08:00:44.397753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:44.397742    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z" event={"ID":"60d7c427-c0a1-4e19-aaeb-5a255674871c","Type":"ContainerDied","Data":"7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7"}
Oct 30 08:00:44.398006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:44.397791    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b1da9239c7015376a73a3937caf9360342b3143fdb047473104bfa122295bd7"
Oct 30 08:00:44.699084 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:44.698992    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:46.571369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:46.571325    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:47.759174 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759137    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"]
Oct 30 08:00:47.759882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759866    6738 cpu_manager.go:414] "containerMap: removing container" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759883    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759905    6738 cpu_manager.go:414] "containerMap: removing container" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759911    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57826627-652f-4b1b-a66a-84d6075e13a4" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759936    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759941    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5b65bd-7e5e-4d28-988a-dd2035b41572" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759951    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:47.759955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759956    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759963    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759967    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759977    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759983    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd7ba37-da03-4528-a350-c4973aa9f9f4" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759989    6738 cpu_manager.go:414] "containerMap: removing container" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.759993    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760001    6738 cpu_manager.go:414] "containerMap: removing container" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760006    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760015    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760018    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5232edb-fed4-493e-96f9-077bba327cca" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760038    6738 cpu_manager.go:414] "containerMap: removing container" podUID="60d7c427-c0a1-4e19-aaeb-5a255674871c" containerName="revoke-keys"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760042    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d7c427-c0a1-4e19-aaeb-5a255674871c" containerName="revoke-keys"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760149    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48710cf-4d3c-4d1c-a0dd-893da16b8bf6" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760163    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="61865956-c60b-471e-a066-33cbee6cd28c" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760171    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e81ec-5cb1-4b49-a878-e5f170457614" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760177    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3bab7-4d4d-4d0a-b4f0-e66f99b7be32" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760185    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d904f9f-9c80-4d8d-a0be-13c1b179afe1" containerName="cleanup"
Oct 30 08:00:47.760226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.760193    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d7c427-c0a1-4e19-aaeb-5a255674871c" containerName="revoke-keys"
Oct 30 08:00:47.764010 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.763987    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.780198 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.780168    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:00:47.792762 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.792723    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:00:47.804774 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.804725    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"]
Oct 30 08:00:47.889238 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.889203    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.889446 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.889276    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.889504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.889469    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9l9m\" (UniqueName: \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.991332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.991256    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.991527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.991384    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.991527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.991474    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9l9m\" (UniqueName: \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.992081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.992052    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:47.993857 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:47.993825    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:48.019101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.019032    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9l9m\" (UniqueName: \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:48.067273 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.067239    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"]
Oct 30 08:00:48.073874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.073844    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec983ff7-sbbjt"]
Oct 30 08:00:48.079262 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.079231    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:48.364403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.363021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:48.403929 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.403896    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"]
Oct 30 08:00:48.404538 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:00:48.404509    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5805a2af_5647_471f_85bf_5d2d10631fb3.slice/crio-a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a WatchSource:0}: Error finding container a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a: Status 404 returned error can't find the container with id a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a
Oct 30 08:00:48.430366 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.430327    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk" event={"ID":"5805a2af-5647-471f-85bf-5d2d10631fb3","Type":"ContainerStarted","Data":"a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a"}
Oct 30 08:00:48.671757 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.671609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:48.702186 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.702151    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 08:00:48.702459 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.702430    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy" containerID="cri-o://391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53" gracePeriod=30
Oct 30 08:00:48.850683 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:48.850650    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:49.035057 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.034963    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:49.295591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.295499    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"]
Oct 30 08:00:49.301013 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.300987    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.320211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.320181    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"]
Oct 30 08:00:49.427568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.427531    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.427744 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.427617    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8qw\" (UniqueName: \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.427744 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.427663    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.440455 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.440419    6738 generic.go:356] "Generic (PLEG): container finished" podID="5805a2af-5647-471f-85bf-5d2d10631fb3" containerID="3831dcd58f59481ddb82088915adb681bb006cbb6285ace4d2229c4e3b2f3226" exitCode=0
Oct 30 08:00:49.440641 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.440504    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk" event={"ID":"5805a2af-5647-471f-85bf-5d2d10631fb3","Type":"ContainerDied","Data":"3831dcd58f59481ddb82088915adb681bb006cbb6285ace4d2229c4e3b2f3226"}
Oct 30 08:00:49.529027 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.528991    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.529216 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.529076    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8qw\" (UniqueName: \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.529216 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.529108    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.529857 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.529832    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.531469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.531447    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.552973 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.552901    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8qw\" (UniqueName: \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.618244 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.618204    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:49.705515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.701533    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc97af42-5355-495c-b5bf-4bc52759ad46" path="/var/lib/kubelet/pods/cc97af42-5355-495c-b5bf-4bc52759ad46/volumes"
Oct 30 08:00:49.950227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:49.950193    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"]
Oct 30 08:00:49.950737 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:00:49.950712    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1028d367_ac06_421d_8c48_58753013e976.slice/crio-ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e WatchSource:0}: Error finding container ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e: Status 404 returned error can't find the container with id ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e
Oct 30 08:00:50.451278 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.450959    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg" event={"ID":"1028d367-ac06-421d-8c48-58753013e976","Type":"ContainerStarted","Data":"995b787c28a2634c47369284526f05260646df79712b98c5ad9b2f703cbea545"}
Oct 30 08:00:50.451278 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.451023    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg" event={"ID":"1028d367-ac06-421d-8c48-58753013e976","Type":"ContainerStarted","Data":"ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e"}
Oct 30 08:00:50.494209 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.494154    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg" podStartSLOduration=1.494137802 podStartE2EDuration="1.494137802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:50.49164853 +0000 UTC m=+3810.924412425" watchObservedRunningTime="2026-10-30 08:00:50.494137802 +0000 UTC m=+3810.926901695"
Oct 30 08:00:50.614867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.614832    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:50.746179 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.746141    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\") pod \"5805a2af-5647-471f-85bf-5d2d10631fb3\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") "
Oct 30 08:00:50.746438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.746246    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\") pod \"5805a2af-5647-471f-85bf-5d2d10631fb3\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") "
Oct 30 08:00:50.746438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.746360    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\" (UniqueName: \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\") pod \"5805a2af-5647-471f-85bf-5d2d10631fb3\" (UID: \"5805a2af-5647-471f-85bf-5d2d10631fb3\") "
Oct 30 08:00:50.746729 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.746705    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca" pod "5805a2af-5647-471f-85bf-5d2d10631fb3" (UID: "5805a2af-5647-471f-85bf-5d2d10631fb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:50.747055 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.747024    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:50.748735 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.748703    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m" pod "5805a2af-5647-471f-85bf-5d2d10631fb3" (UID: "5805a2af-5647-471f-85bf-5d2d10631fb3"). InnerVolumeSpecName "kube-api-access-t9l9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:50.748866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.748840    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca" pod "5805a2af-5647-471f-85bf-5d2d10631fb3" (UID: "5805a2af-5647-471f-85bf-5d2d10631fb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:50.848771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.848680    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9l9m\" (UniqueName: \"kubernetes.io/projected/5805a2af-5647-471f-85bf-5d2d10631fb3-kube-api-access-t9l9m\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:50.848771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:50.848715    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5805a2af-5647-471f-85bf-5d2d10631fb3-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:51.045639 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.045594    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:51.367678 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.367642    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:51.490336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.483276    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk" event={"ID":"5805a2af-5647-471f-85bf-5d2d10631fb3","Type":"ContainerDied","Data":"a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a"}
Oct 30 08:00:51.490336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.483390    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35194a899834ba095bf75ed92163df0230a9642ad39643b1aaa9df96440083a"
Oct 30 08:00:51.490336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.483458    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"
Oct 30 08:00:51.490336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.487971    6738 generic.go:356] "Generic (PLEG): container finished" podID="1028d367-ac06-421d-8c48-58753013e976" containerID="995b787c28a2634c47369284526f05260646df79712b98c5ad9b2f703cbea545" exitCode=0
Oct 30 08:00:51.490336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.488208    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg" event={"ID":"1028d367-ac06-421d-8c48-58753013e976","Type":"ContainerDied","Data":"995b787c28a2634c47369284526f05260646df79712b98c5ad9b2f703cbea545"}
Oct 30 08:00:51.533511 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:51.533474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:52.651561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.651532    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:52.786125 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.786027    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\" (UniqueName: \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\") pod \"1028d367-ac06-421d-8c48-58753013e976\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") "
Oct 30 08:00:52.786125 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.786077    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\") pod \"1028d367-ac06-421d-8c48-58753013e976\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") "
Oct 30 08:00:52.786457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.786216    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\") pod \"1028d367-ac06-421d-8c48-58753013e976\" (UID: \"1028d367-ac06-421d-8c48-58753013e976\") "
Oct 30 08:00:52.786594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.786566    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca" pod "1028d367-ac06-421d-8c48-58753013e976" (UID: "1028d367-ac06-421d-8c48-58753013e976"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:52.787361 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.787047    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1028d367-ac06-421d-8c48-58753013e976-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:52.788561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.788533    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca" pod "1028d367-ac06-421d-8c48-58753013e976" (UID: "1028d367-ac06-421d-8c48-58753013e976"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:52.788561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.788544    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw" pod "1028d367-ac06-421d-8c48-58753013e976" (UID: "1028d367-ac06-421d-8c48-58753013e976"). InnerVolumeSpecName "kube-api-access-mg8qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:52.888866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.888830    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8qw\" (UniqueName: \"kubernetes.io/projected/1028d367-ac06-421d-8c48-58753013e976-kube-api-access-mg8qw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:52.888866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:52.888863    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1028d367-ac06-421d-8c48-58753013e976-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:53.186608 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:53.186573    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:53.506258 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:53.506174    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg" event={"ID":"1028d367-ac06-421d-8c48-58753013e976","Type":"ContainerDied","Data":"ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e"}
Oct 30 08:00:53.506258 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:53.506212    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"
Oct 30 08:00:53.506258 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:53.506234    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae671bbdd306894c30bcf7e81165e2f7e113c63e223bbce73efcca6e666b726e"
Oct 30 08:00:53.990945 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:53.990918    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 08:00:54.106170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106073    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106150    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106256    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106306    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106390    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106431    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106458    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106512    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106584    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:00:54.106750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106714    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\") pod \"5b41f38b-61e8-45b8-9647-f91d6674a455\" (UID: \"5b41f38b-61e8-45b8-9647-f91d6674a455\") "
Oct 30 08:00:54.106917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106768    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:00:54.106917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106809    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:00:54.106917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106902    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:00:54.107114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.106990    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:54.107284 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.107264    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.107284 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.107283    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.107476 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.107325    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b41f38b-61e8-45b8-9647-f91d6674a455-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.107476 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.107338    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.107476 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.107346    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.108706 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.108675    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:00:54.108927 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.108906    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "kube-api-access-rcmb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:54.109023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.108932    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:54.109023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.108988    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo" pod "5b41f38b-61e8-45b8-9647-f91d6674a455" (UID: "5b41f38b-61e8-45b8-9647-f91d6674a455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:00:54.208544 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.208507    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.208544 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.208540    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.208544 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.208553    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.208812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.208566    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcmb5\" (UniqueName: \"kubernetes.io/projected/5b41f38b-61e8-45b8-9647-f91d6674a455-kube-api-access-rcmb5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:54.517211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.517176    6738 generic.go:356] "Generic (PLEG): container finished" podID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerID="391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53" exitCode=0
Oct 30 08:00:54.517454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.517247    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"
Oct 30 08:00:54.517454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.517267    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerDied","Data":"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"}
Oct 30 08:00:54.517454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.517347    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl" event={"ID":"5b41f38b-61e8-45b8-9647-f91d6674a455","Type":"ContainerDied","Data":"2eb7f0e4a4c0ec7644b2ca98ef160031c9fe9fd2b467c828229e0de6e9d4a92b"}
Oct 30 08:00:54.517454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.517374    6738 scope.go:122] "RemoveContainer" containerID="391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"
Oct 30 08:00:54.529444 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.529423    6738 scope.go:122] "RemoveContainer" containerID="2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"
Oct 30 08:00:54.541260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.541240    6738 scope.go:122] "RemoveContainer" containerID="391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"
Oct 30 08:00:54.541569 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:54.541545    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53\": container with ID starting with 391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53 not found: ID does not exist" containerID="391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"
Oct 30 08:00:54.541659 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.541582    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53"} err="failed to get container status \"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53\": rpc error: code = NotFound desc = could not find container \"391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53\": container with ID starting with 391fa022bd1e1ed8033efefb3504688e5c4c334eaf92c6459cb5f054369b3d53 not found: ID does not exist"
Oct 30 08:00:54.541659 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.541605    6738 scope.go:122] "RemoveContainer" containerID="2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"
Oct 30 08:00:54.541900 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:00:54.541874    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63\": container with ID starting with 2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63 not found: ID does not exist" containerID="2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"
Oct 30 08:00:54.541942 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.541906    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63"} err="failed to get container status \"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63\": rpc error: code = NotFound desc = could not find container \"2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63\": container with ID starting with 2e2451cf03db3a9596861ec8f6cce2bce12bcbcc4371b2d2462bf7a2c4edcc63 not found: ID does not exist"
Oct 30 08:00:54.573014 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.572974    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 08:00:54.585893 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:54.585854    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f34aa-openshift-default-9fd58b449-zw7fl"]
Oct 30 08:00:55.042993 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:55.042961    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:55.690996 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:55.690950    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" path="/var/lib/kubelet/pods/5b41f38b-61e8-45b8-9647-f91d6674a455/volumes"
Oct 30 08:00:56.959971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:56.959933    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:58.277785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.277750    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 08:00:58.392202 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.392159    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:58.427661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.427621    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 08:00:58.427906 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.427878    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" podUID="45566de8-c39d-4c27-9cf0-3afd7822c95e" containerName="maas-api" containerID="cri-o://8c95d21e3eb5b1c88ca303ee3ca57c01efdf43ee067c23376f65dbe78d9eec44" gracePeriod=30
Oct 30 08:00:58.529730 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.529642    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:58.558556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.558520    6738 generic.go:356] "Generic (PLEG): container finished" podID="45566de8-c39d-4c27-9cf0-3afd7822c95e" containerID="8c95d21e3eb5b1c88ca303ee3ca57c01efdf43ee067c23376f65dbe78d9eec44" exitCode=0
Oct 30 08:00:58.558716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.558588    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" event={"ID":"45566de8-c39d-4c27-9cf0-3afd7822c95e","Type":"ContainerDied","Data":"8c95d21e3eb5b1c88ca303ee3ca57c01efdf43ee067c23376f65dbe78d9eec44"}
Oct 30 08:00:58.700813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.700779    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 08:00:58.773612 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.773568    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\") pod \"45566de8-c39d-4c27-9cf0-3afd7822c95e\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") "
Oct 30 08:00:58.773781 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.773695    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\") pod \"45566de8-c39d-4c27-9cf0-3afd7822c95e\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") "
Oct 30 08:00:58.773781 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.773735    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") pod \"45566de8-c39d-4c27-9cf0-3afd7822c95e\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") "
Oct 30 08:00:58.773927 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.773876    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\" (UniqueName: \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\") pod \"45566de8-c39d-4c27-9cf0-3afd7822c95e\" (UID: \"45566de8-c39d-4c27-9cf0-3afd7822c95e\") "
Oct 30 08:00:58.774166 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.774131    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca" pod "45566de8-c39d-4c27-9cf0-3afd7822c95e" (UID: "45566de8-c39d-4c27-9cf0-3afd7822c95e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:58.774852 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.774827    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45566de8-c39d-4c27-9cf0-3afd7822c95e-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:58.776225 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.776195    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls" pod "45566de8-c39d-4c27-9cf0-3afd7822c95e" (UID: "45566de8-c39d-4c27-9cf0-3afd7822c95e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:58.776473 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.776448    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs" pod "45566de8-c39d-4c27-9cf0-3afd7822c95e" (UID: "45566de8-c39d-4c27-9cf0-3afd7822c95e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:58.777118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.777084    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh" pod "45566de8-c39d-4c27-9cf0-3afd7822c95e" (UID: "45566de8-c39d-4c27-9cf0-3afd7822c95e"). InnerVolumeSpecName "kube-api-access-kc9zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:58.880860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.880821    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:58.880860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.880855    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9zh\" (UniqueName: \"kubernetes.io/projected/45566de8-c39d-4c27-9cf0-3afd7822c95e-kube-api-access-kc9zh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:58.880860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:58.880864    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45566de8-c39d-4c27-9cf0-3afd7822c95e-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:00:59.355603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.355568    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:59.569208 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.569178    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"
Oct 30 08:00:59.569408 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.569208    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b" event={"ID":"45566de8-c39d-4c27-9cf0-3afd7822c95e","Type":"ContainerDied","Data":"a81118683a4fffaa0311564e4be5c67b190162d32c1947f78f07ac95c0ed43fb"}
Oct 30 08:00:59.569408 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.569274    6738 scope.go:122] "RemoveContainer" containerID="8c95d21e3eb5b1c88ca303ee3ca57c01efdf43ee067c23376f65dbe78d9eec44"
Oct 30 08:00:59.582499 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.582474    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 08:00:59.671747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.671712    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 08:00:59.678394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.678358    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8e7948-7878f55c55-xnp6b"]
Oct 30 08:00:59.695187 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.695149    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45566de8-c39d-4c27-9cf0-3afd7822c95e" path="/var/lib/kubelet/pods/45566de8-c39d-4c27-9cf0-3afd7822c95e/volumes"
Oct 30 08:00:59.697473 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.697443    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:59.830217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.830181    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:00:59.869368 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.869339    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"]
Oct 30 08:00:59.872819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.872782    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8e7948-vzkxk"]
Oct 30 08:00:59.958403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:00:59.956428    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:00.389352 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:00.387254    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:01.523320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:01.517586    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:01.691923 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:01.691885    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5805a2af-5647-471f-85bf-5d2d10631fb3" path="/var/lib/kubelet/pods/5805a2af-5647-471f-85bf-5d2d10631fb3/volumes"
Oct 30 08:01:03.279017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:03.278973    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing" containerID="cri-o://baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537" gracePeriod=25
Oct 30 08:01:03.314943 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:03.314895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:04.582816 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.582765    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" podUID="a07b0829-804f-4ef2-929e-482c02ca641c" containerName="payload-pre-processing" containerID="cri-o://4e6e98850a8f57a284cb4ba687ea4a89521d134f819764b1c7246ce55eca9d4a" gracePeriod=25
Oct 30 08:01:04.803337 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.801422    6738 generic.go:356] "Generic (PLEG): container finished" podID="a07b0829-804f-4ef2-929e-482c02ca641c" containerID="4e6e98850a8f57a284cb4ba687ea4a89521d134f819764b1c7246ce55eca9d4a" exitCode=0
Oct 30 08:01:04.803337 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.801670    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" event={"ID":"a07b0829-804f-4ef2-929e-482c02ca641c","Type":"ContainerDied","Data":"4e6e98850a8f57a284cb4ba687ea4a89521d134f819764b1c7246ce55eca9d4a"}
Oct 30 08:01:04.871752 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.871726    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 08:01:04.964962 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.964927    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:04.984038 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.984002    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\" (UniqueName: \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\") pod \"a07b0829-804f-4ef2-929e-482c02ca641c\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") "
Oct 30 08:01:04.984220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.984204    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\") pod \"a07b0829-804f-4ef2-929e-482c02ca641c\" (UID: \"a07b0829-804f-4ef2-929e-482c02ca641c\") "
Oct 30 08:01:04.984737 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.984708    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume" pod "a07b0829-804f-4ef2-929e-482c02ca641c" (UID: "a07b0829-804f-4ef2-929e-482c02ca641c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:04.984888 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.984859    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07b0829-804f-4ef2-929e-482c02ca641c-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:04.986332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:04.986307    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64" pod "a07b0829-804f-4ef2-929e-482c02ca641c" (UID: "a07b0829-804f-4ef2-929e-482c02ca641c"). InnerVolumeSpecName "kube-api-access-twb64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:05.086163 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.086120    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twb64\" (UniqueName: \"kubernetes.io/projected/a07b0829-804f-4ef2-929e-482c02ca641c-kube-api-access-twb64\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:05.446533 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.446488    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:05.810645 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.810601    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs" event={"ID":"a07b0829-804f-4ef2-929e-482c02ca641c","Type":"ContainerDied","Data":"32e92936f52236c5f9868efed1c6b43231ee47d0a63bd8f034d5fb43e93e3e5b"}
Oct 30 08:01:05.810645 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.810630    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"
Oct 30 08:01:05.811138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.810674    6738 scope.go:122] "RemoveContainer" containerID="4e6e98850a8f57a284cb4ba687ea4a89521d134f819764b1c7246ce55eca9d4a"
Oct 30 08:01:05.866710 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.866670    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 08:01:05.879432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:05.879397    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a475d6-6fcd448759-m7pxs"]
Oct 30 08:01:07.129856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.129821    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"]
Oct 30 08:01:07.130843 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130818    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1028d367-ac06-421d-8c48-58753013e976" containerName="revoke-keys"
Oct 30 08:01:07.130843 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130843    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1028d367-ac06-421d-8c48-58753013e976" containerName="revoke-keys"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130877    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130892    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130933    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5805a2af-5647-471f-85bf-5d2d10631fb3" containerName="revoke-keys"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.130941    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5805a2af-5647-471f-85bf-5d2d10631fb3" containerName="revoke-keys"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131001    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a07b0829-804f-4ef2-929e-482c02ca641c" containerName="payload-pre-processing"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131011    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07b0829-804f-4ef2-929e-482c02ca641c" containerName="payload-pre-processing"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131023    6738 cpu_manager.go:414] "containerMap: removing container" podUID="45566de8-c39d-4c27-9cf0-3afd7822c95e" containerName="maas-api"
Oct 30 08:01:07.131041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131030    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45566de8-c39d-4c27-9cf0-3afd7822c95e" containerName="maas-api"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131215    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131235    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131247    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5805a2af-5647-471f-85bf-5d2d10631fb3" containerName="revoke-keys"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131272    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07b0829-804f-4ef2-929e-482c02ca641c" containerName="payload-pre-processing"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131282    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="45566de8-c39d-4c27-9cf0-3afd7822c95e" containerName="maas-api"
Oct 30 08:01:07.131549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.131311    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1028d367-ac06-421d-8c48-58753013e976" containerName="revoke-keys"
Oct 30 08:01:07.134428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.134401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:07.136150 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.136131    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.152847 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.152818    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"]
Oct 30 08:01:07.160519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.160487    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:07.160519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.160507    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:07.221769 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.221717    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.226191 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.226162    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.226506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.226240    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpct\" (UniqueName: \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.253378 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.253340    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:07.365867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.330437    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.365867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.330547    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpct\" (UniqueName: \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.365867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.330797    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.365867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.331840    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.365867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.334732    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.387574 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.380495    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpct\" (UniqueName: \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\") pod \"maas-api-revoke-sub-1da4ac29-hfxbt\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.416680 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.416646    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:07.451752 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.451714    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:07.529524 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:07.529487    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:01:07.583994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.575926    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 08:01:07.583994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.576319    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy" containerID="cri-o://a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5" gracePeriod=30
Oct 30 08:01:07.629174 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.629121    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:07.771477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.771274    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07b0829-804f-4ef2-929e-482c02ca641c" path="/var/lib/kubelet/pods/a07b0829-804f-4ef2-929e-482c02ca641c/volumes"
Oct 30 08:01:07.837884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:07.837854    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"]
Oct 30 08:01:08.043775 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.043690    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"]
Oct 30 08:01:08.044834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.044813    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:08.044834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.044835    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b41f38b-61e8-45b8-9647-f91d6674a455" containerName="istio-proxy"
Oct 30 08:01:08.050567 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.050544    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.068506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.068469    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"]
Oct 30 08:01:08.142902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.142863    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.143430 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.142934    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmq8g\" (UniqueName: \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.143430 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.143209    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.244706 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.244665    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.244909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.244767    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmq8g\" (UniqueName: \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.244965 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.244950    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.245602 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.245576    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.247161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.247141    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.270848 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.270824    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmq8g\" (UniqueName: \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.368170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.368076    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:08.677273 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.677241    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"]
Oct 30 08:01:08.678523 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:08.678486    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af5f6ee_7f5f_4ec4_a326_9818a29a1911.slice/crio-1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d WatchSource:0}: Error finding container 1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d: Status 404 returned error can't find the container with id 1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d
Oct 30 08:01:08.839001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.838962    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd" event={"ID":"8af5f6ee-7f5f-4ec4-a326-9818a29a1911","Type":"ContainerStarted","Data":"1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d"}
Oct 30 08:01:08.840874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.840837    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt" event={"ID":"cbd196d7-e2ee-415d-bff1-daa621aca270","Type":"ContainerStarted","Data":"570cda668506b1b4fdcf17d595f33d9a63141cf8e05920825a29dc14d645fc54"}
Oct 30 08:01:08.841013 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.840895    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt" event={"ID":"cbd196d7-e2ee-415d-bff1-daa621aca270","Type":"ContainerStarted","Data":"039e530e86485f7bebd0f08b7398a1ee09aa7ffff2ce7659a72b9f005c30d321"}
Oct 30 08:01:08.884780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:08.884717    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt" podStartSLOduration=1.8846998 podStartE2EDuration="1.8846998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:08.882088747 +0000 UTC m=+3829.314852639" watchObservedRunningTime="2026-10-30 08:01:08.8846998 +0000 UTC m=+3829.317463692"
Oct 30 08:01:09.322916 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:09.322876    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:09.851168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:09.851132    6738 generic.go:356] "Generic (PLEG): container finished" podID="cbd196d7-e2ee-415d-bff1-daa621aca270" containerID="570cda668506b1b4fdcf17d595f33d9a63141cf8e05920825a29dc14d645fc54" exitCode=0
Oct 30 08:01:09.851381 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:09.851208    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt" event={"ID":"cbd196d7-e2ee-415d-bff1-daa621aca270","Type":"ContainerDied","Data":"570cda668506b1b4fdcf17d595f33d9a63141cf8e05920825a29dc14d645fc54"}
Oct 30 08:01:09.852908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:09.852880    6738 generic.go:356] "Generic (PLEG): container finished" podID="8af5f6ee-7f5f-4ec4-a326-9818a29a1911" containerID="02023c95400dbad8d4c4fe9616a16712239b0daf90df5fc0281957c0a7e9d21f" exitCode=0
Oct 30 08:01:09.853030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:09.852927    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd" event={"ID":"8af5f6ee-7f5f-4ec4-a326-9818a29a1911","Type":"ContainerDied","Data":"02023c95400dbad8d4c4fe9616a16712239b0daf90df5fc0281957c0a7e9d21f"}
Oct 30 08:01:10.535529 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.535488    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"]
Oct 30 08:01:10.540060 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.540039    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.560376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.560339    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"]
Oct 30 08:01:10.681749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.681714    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2vl\" (UniqueName: \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.681959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.681762    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.682029 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.681979    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.783808 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.783741    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.783986 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.783937    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2vl\" (UniqueName: \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.784083 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.783987    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.784519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.784498    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.786502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.786445    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.807843 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.807810    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2vl\" (UniqueName: \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\") pod \"maas-api-revoke-sub-f77452d0-fgdf6\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:10.853946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:10.853908    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:11.121239 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.121207    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:11.123219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.123199    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:11.183495 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:11.183455    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73626942_0d27_44a8_9d5d_13d4ce96404f.slice/crio-b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45 WatchSource:0}: Error finding container b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45: Status 404 returned error can't find the container with id b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45
Oct 30 08:01:11.188630 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.188595    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"]
Oct 30 08:01:11.191690 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.191660    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\") pod \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") "
Oct 30 08:01:11.191800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.191710    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\") pod \"cbd196d7-e2ee-415d-bff1-daa621aca270\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") "
Oct 30 08:01:11.191850 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.191815    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\" (UniqueName: \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\") pod \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") "
Oct 30 08:01:11.191909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.191858    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\") pod \"cbd196d7-e2ee-415d-bff1-daa621aca270\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") "
Oct 30 08:01:11.192032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192013    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\") pod \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\" (UID: \"8af5f6ee-7f5f-4ec4-a326-9818a29a1911\") "
Oct 30 08:01:11.192131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192060    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\" (UniqueName: \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\") pod \"cbd196d7-e2ee-415d-bff1-daa621aca270\" (UID: \"cbd196d7-e2ee-415d-bff1-daa621aca270\") "
Oct 30 08:01:11.192195 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192138    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca" pod "cbd196d7-e2ee-415d-bff1-daa621aca270" (UID: "cbd196d7-e2ee-415d-bff1-daa621aca270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:11.192520 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192489    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca" pod "8af5f6ee-7f5f-4ec4-a326-9818a29a1911" (UID: "8af5f6ee-7f5f-4ec4-a326-9818a29a1911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:11.192768 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192733    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.192869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.192772    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.194534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.194507    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g" pod "8af5f6ee-7f5f-4ec4-a326-9818a29a1911" (UID: "8af5f6ee-7f5f-4ec4-a326-9818a29a1911"). InnerVolumeSpecName "kube-api-access-kmq8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:11.194873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.194829    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca" pod "cbd196d7-e2ee-415d-bff1-daa621aca270" (UID: "cbd196d7-e2ee-415d-bff1-daa621aca270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:11.195129 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.195106    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca" pod "8af5f6ee-7f5f-4ec4-a326-9818a29a1911" (UID: "8af5f6ee-7f5f-4ec4-a326-9818a29a1911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:11.195595 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.195571    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct" pod "cbd196d7-e2ee-415d-bff1-daa621aca270" (UID: "cbd196d7-e2ee-415d-bff1-daa621aca270"). InnerVolumeSpecName "kube-api-access-jlpct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:11.294460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.294423    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd196d7-e2ee-415d-bff1-daa621aca270-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.294460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.294455    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpct\" (UniqueName: \"kubernetes.io/projected/cbd196d7-e2ee-415d-bff1-daa621aca270-kube-api-access-jlpct\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.294705 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.294471    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.294705 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.294480    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmq8g\" (UniqueName: \"kubernetes.io/projected/8af5f6ee-7f5f-4ec4-a326-9818a29a1911-kube-api-access-kmq8g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:11.357347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.357245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:11.872971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.872939    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"
Oct 30 08:01:11.872971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.872951    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt" event={"ID":"cbd196d7-e2ee-415d-bff1-daa621aca270","Type":"ContainerDied","Data":"039e530e86485f7bebd0f08b7398a1ee09aa7ffff2ce7659a72b9f005c30d321"}
Oct 30 08:01:11.873509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.873002    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039e530e86485f7bebd0f08b7398a1ee09aa7ffff2ce7659a72b9f005c30d321"
Oct 30 08:01:11.874745 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.874725    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"
Oct 30 08:01:11.874885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.874765    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd" event={"ID":"8af5f6ee-7f5f-4ec4-a326-9818a29a1911","Type":"ContainerDied","Data":"1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d"}
Oct 30 08:01:11.874885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.874810    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f44d01a299750386a9fbcd61af81211bdd344aee1ab22b14a3c417fbabe190d"
Oct 30 08:01:11.876778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.876736    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6" event={"ID":"73626942-0d27-44a8-9d5d-13d4ce96404f","Type":"ContainerStarted","Data":"14ab2cd48e176d8bb88704e7625fca77e6580ea26c8d5d3799ed72a568b94245"}
Oct 30 08:01:11.876929 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.876791    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6" event={"ID":"73626942-0d27-44a8-9d5d-13d4ce96404f","Type":"ContainerStarted","Data":"b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45"}
Oct 30 08:01:11.919195 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:11.919121    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6" podStartSLOduration=1.9191025229999998 podStartE2EDuration="1.919102523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:11.917543918 +0000 UTC m=+3832.350307812" watchObservedRunningTime="2026-10-30 08:01:11.919102523 +0000 UTC m=+3832.351866417"
Oct 30 08:01:12.877928 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.877899    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 08:01:12.888320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.888262    6738 generic.go:356] "Generic (PLEG): container finished" podID="95bf58b8-3623-494a-bb85-1b013279b919" containerID="a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5" exitCode=0
Oct 30 08:01:12.888494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.888334    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerDied","Data":"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"}
Oct 30 08:01:12.888494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.888397    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx" event={"ID":"95bf58b8-3623-494a-bb85-1b013279b919","Type":"ContainerDied","Data":"91e7a128d6b9f0f0975195f6bb142f88775c9ab826882efe76db68d8174e5c88"}
Oct 30 08:01:12.888494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.888348    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"
Oct 30 08:01:12.888494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.888425    6738 scope.go:122] "RemoveContainer" containerID="a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"
Oct 30 08:01:12.890375 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.890347    6738 generic.go:356] "Generic (PLEG): container finished" podID="73626942-0d27-44a8-9d5d-13d4ce96404f" containerID="14ab2cd48e176d8bb88704e7625fca77e6580ea26c8d5d3799ed72a568b94245" exitCode=0
Oct 30 08:01:12.890527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.890431    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6" event={"ID":"73626942-0d27-44a8-9d5d-13d4ce96404f","Type":"ContainerDied","Data":"14ab2cd48e176d8bb88704e7625fca77e6580ea26c8d5d3799ed72a568b94245"}
Oct 30 08:01:12.899728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.899702    6738 scope.go:122] "RemoveContainer" containerID="e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"
Oct 30 08:01:12.912675 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.912651    6738 scope.go:122] "RemoveContainer" containerID="a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"
Oct 30 08:01:12.913089 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:12.913068    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5\": container with ID starting with a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5 not found: ID does not exist" containerID="a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"
Oct 30 08:01:12.913196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.913103    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5"} err="failed to get container status \"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5\": rpc error: code = NotFound desc = could not find container \"a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5\": container with ID starting with a5d8680b1e055d01d691db2a7b3c1629beb4e40efd39932480e8886966ff11c5 not found: ID does not exist"
Oct 30 08:01:12.913196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.913130    6738 scope.go:122] "RemoveContainer" containerID="e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"
Oct 30 08:01:12.913436 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:12.913419    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384\": container with ID starting with e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384 not found: ID does not exist" containerID="e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"
Oct 30 08:01:12.913504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:12.913445    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384"} err="failed to get container status \"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384\": rpc error: code = NotFound desc = could not find container \"e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384\": container with ID starting with e7c9d61c6e7bd693a772aeff1a8f27d17cf3007c465b3918c4c12a649755e384 not found: ID does not exist"
Oct 30 08:01:13.043564 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043526    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043608    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043651    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043688    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043732    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043874    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043904    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.043954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043924    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:13.043954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.043948    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.044568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044010    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\") pod \"95bf58b8-3623-494a-bb85-1b013279b919\" (UID: \"95bf58b8-3623-494a-bb85-1b013279b919\") "
Oct 30 08:01:13.044568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044105    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:13.044568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044165    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:13.044568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:13.044568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044405    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:13.044878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044862    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.044940 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044886    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.044940 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044902    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.044940 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044917    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bf58b8-3623-494a-bb85-1b013279b919-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.044940 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.044932    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.046019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.045993    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:13.046155 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.046031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:13.046155 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.046053    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "kube-api-access-zg8nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:13.046397 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.046380    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo" pod "95bf58b8-3623-494a-bb85-1b013279b919" (UID: "95bf58b8-3623-494a-bb85-1b013279b919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:01:13.145885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.145846    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bf58b8-3623-494a-bb85-1b013279b919-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.145885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.145879    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg8nm\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-kube-api-access-zg8nm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.145885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.145889    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bf58b8-3623-494a-bb85-1b013279b919-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.146150 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.145900    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bf58b8-3623-494a-bb85-1b013279b919-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:13.239957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.239919    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 08:01:13.246457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.246424    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8e7948-openshift-default-584c499449-4zpkx"]
Oct 30 08:01:13.255607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.255574    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:13.692106 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:13.692068    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95bf58b8-3623-494a-bb85-1b013279b919" path="/var/lib/kubelet/pods/95bf58b8-3623-494a-bb85-1b013279b919/volumes"
Oct 30 08:01:14.059839 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.059807    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:14.159432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.159399    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\") pod \"73626942-0d27-44a8-9d5d-13d4ce96404f\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") "
Oct 30 08:01:14.159667 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.159584    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\" (UniqueName: \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\") pod \"73626942-0d27-44a8-9d5d-13d4ce96404f\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") "
Oct 30 08:01:14.159747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.159694    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\") pod \"73626942-0d27-44a8-9d5d-13d4ce96404f\" (UID: \"73626942-0d27-44a8-9d5d-13d4ce96404f\") "
Oct 30 08:01:14.160111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.160080    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca" pod "73626942-0d27-44a8-9d5d-13d4ce96404f" (UID: "73626942-0d27-44a8-9d5d-13d4ce96404f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:14.160315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.160276    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:14.161697 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.161677    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl" pod "73626942-0d27-44a8-9d5d-13d4ce96404f" (UID: "73626942-0d27-44a8-9d5d-13d4ce96404f"). InnerVolumeSpecName "kube-api-access-8f2vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:14.161767 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.161739    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca" pod "73626942-0d27-44a8-9d5d-13d4ce96404f" (UID: "73626942-0d27-44a8-9d5d-13d4ce96404f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:14.261279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.261184    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2vl\" (UniqueName: \"kubernetes.io/projected/73626942-0d27-44a8-9d5d-13d4ce96404f-kube-api-access-8f2vl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:14.261279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.261222    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73626942-0d27-44a8-9d5d-13d4ce96404f-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:14.908949 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.908908    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"
Oct 30 08:01:14.909158 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.908909    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6" event={"ID":"73626942-0d27-44a8-9d5d-13d4ce96404f","Type":"ContainerDied","Data":"b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45"}
Oct 30 08:01:14.909158 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.909049    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b31cd381c9e2aeae4139bdc7917e5bc029d9a1bbbf693bfc4a1b193d78fb9c45"
Oct 30 08:01:14.965277 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:14.965244    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:15.033855 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:15.033806    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:16.817223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:16.817186    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:17.474365 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.474328    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"]
Oct 30 08:01:17.475128 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475107    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cbd196d7-e2ee-415d-bff1-daa621aca270" containerName="revoke-keys"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475130    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd196d7-e2ee-415d-bff1-daa621aca270" containerName="revoke-keys"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475152    6738 cpu_manager.go:414] "containerMap: removing container" podUID="73626942-0d27-44a8-9d5d-13d4ce96404f" containerName="revoke-keys"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475161    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73626942-0d27-44a8-9d5d-13d4ce96404f" containerName="revoke-keys"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475185    6738 cpu_manager.go:414] "containerMap: removing container" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475193    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475228    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8af5f6ee-7f5f-4ec4-a326-9818a29a1911" containerName="revoke-keys"
Oct 30 08:01:17.475265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475234    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af5f6ee-7f5f-4ec4-a326-9818a29a1911" containerName="revoke-keys"
Oct 30 08:01:17.475712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475441    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd196d7-e2ee-415d-bff1-daa621aca270" containerName="revoke-keys"
Oct 30 08:01:17.475712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475462    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="73626942-0d27-44a8-9d5d-13d4ce96404f" containerName="revoke-keys"
Oct 30 08:01:17.475712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475477    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:17.475712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475492    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af5f6ee-7f5f-4ec4-a326-9818a29a1911" containerName="revoke-keys"
Oct 30 08:01:17.475712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.475502    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:17.479751 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.479731    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.496191 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.496159    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"]
Oct 30 08:01:17.504142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.504111    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:17.504339 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.504149    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:17.515351 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.515315    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.515513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.515367    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.515611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.515583    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n579p\" (UniqueName: \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.617154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.617116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.617154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.617158    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.617472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.617221    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n579p\" (UniqueName: \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.617984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.617958    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.619658 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.619634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.641367 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.641332    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n579p\" (UniqueName: \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\") pod \"maas-api-revoke-sub-665f3a71-x42wm\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:17.795993 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:17.795895    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:18.110576 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:18.110539    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a431fec_7a10_4470_861a_ccb6092e6fd4.slice/crio-189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1 WatchSource:0}: Error finding container 189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1: Status 404 returned error can't find the container with id 189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1
Oct 30 08:01:18.111384 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:18.111358    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"]
Oct 30 08:01:18.667898 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:18.667864    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:18.802340 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:18.802213    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:18.960956 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:18.960908    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm" event={"ID":"8a431fec-7a10-4470-861a-ccb6092e6fd4","Type":"ContainerStarted","Data":"ef29b1531b3e488295f4985b101892f9a6518628b68a759f761ca2848993f6e6"}
Oct 30 08:01:18.961175 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:18.960984    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm" event={"ID":"8a431fec-7a10-4470-861a-ccb6092e6fd4","Type":"ContainerStarted","Data":"189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1"}
Oct 30 08:01:19.003442 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:19.003389    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm" podStartSLOduration=2.00337357 podStartE2EDuration="2.00337357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:19.000953796 +0000 UTC m=+3839.433717726" watchObservedRunningTime="2026-10-30 08:01:19.00337357 +0000 UTC m=+3839.436137497"
Oct 30 08:01:19.706076 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:19.706040    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:19.970533 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:19.970445    6738 generic.go:356] "Generic (PLEG): container finished" podID="8a431fec-7a10-4470-861a-ccb6092e6fd4" containerID="ef29b1531b3e488295f4985b101892f9a6518628b68a759f761ca2848993f6e6" exitCode=0
Oct 30 08:01:19.970694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:19.970532    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm" event={"ID":"8a431fec-7a10-4470-861a-ccb6092e6fd4","Type":"ContainerDied","Data":"ef29b1531b3e488295f4985b101892f9a6518628b68a759f761ca2848993f6e6"}
Oct 30 08:01:20.401822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.401779    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:20.869106 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.869012    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"]
Oct 30 08:01:20.869966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.869940    6738 cpu_manager.go:414] "containerMap: removing container" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:20.870063 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.869968    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bf58b8-3623-494a-bb85-1b013279b919" containerName="istio-proxy"
Oct 30 08:01:20.874085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.874068    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:20.895426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.895389    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"]
Oct 30 08:01:20.999070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.999026    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqqq\" (UniqueName: \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:20.999279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.999258    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:20.999378 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:20.999357    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.100918 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.100877    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.101124 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.100939    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.101124 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.101085    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqqq\" (UniqueName: \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.101825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.101796    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.103375 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.103352    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.125471 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.125436    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqqq\" (UniqueName: \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\") pod \"maas-api-revoke-sub-984a0e4a-shvpb\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.150335 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.150285    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:21.189142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.189105    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:21.202415 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.202380    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\") pod \"8a431fec-7a10-4470-861a-ccb6092e6fd4\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") "
Oct 30 08:01:21.202600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.202515    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\") pod \"8a431fec-7a10-4470-861a-ccb6092e6fd4\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") "
Oct 30 08:01:21.202697 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.202637    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\" (UniqueName: \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\") pod \"8a431fec-7a10-4470-861a-ccb6092e6fd4\" (UID: \"8a431fec-7a10-4470-861a-ccb6092e6fd4\") "
Oct 30 08:01:21.202807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.202782    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca" pod "8a431fec-7a10-4470-861a-ccb6092e6fd4" (UID: "8a431fec-7a10-4470-861a-ccb6092e6fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:21.203417 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.203388    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:21.205527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.205495    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca" pod "8a431fec-7a10-4470-861a-ccb6092e6fd4" (UID: "8a431fec-7a10-4470-861a-ccb6092e6fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:21.206012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.205994    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p" pod "8a431fec-7a10-4470-861a-ccb6092e6fd4" (UID: "8a431fec-7a10-4470-861a-ccb6092e6fd4"). InnerVolumeSpecName "kube-api-access-n579p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:21.304569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.304534    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n579p\" (UniqueName: \"kubernetes.io/projected/8a431fec-7a10-4470-861a-ccb6092e6fd4-kube-api-access-n579p\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:21.304569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.304570    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a431fec-7a10-4470-861a-ccb6092e6fd4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:21.506980 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:21.506948    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3905c7_e765_4bbe_8422_80c740069251.slice/crio-3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860 WatchSource:0}: Error finding container 3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860: Status 404 returned error can't find the container with id 3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860
Oct 30 08:01:21.507371 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.507343    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"]
Oct 30 08:01:21.991840 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.991807    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"
Oct 30 08:01:21.991840 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.991825    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm" event={"ID":"8a431fec-7a10-4470-861a-ccb6092e6fd4","Type":"ContainerDied","Data":"189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1"}
Oct 30 08:01:21.992274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.991880    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="189808480a4bc1704601d028177114c07ee859ce0118553bab509a5b9cdb28f1"
Oct 30 08:01:21.994089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.993989    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb" event={"ID":"bd3905c7-e765-4bbe-8422-80c740069251","Type":"ContainerStarted","Data":"b07d910d98f3d514e642866663c6386b7a7701ac30f4a01ffee00f1da59cad61"}
Oct 30 08:01:21.994089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:21.994050    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb" event={"ID":"bd3905c7-e765-4bbe-8422-80c740069251","Type":"ContainerStarted","Data":"3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860"}
Oct 30 08:01:22.037682 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:22.037562    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb" podStartSLOduration=2.037547759 podStartE2EDuration="2.037547759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:22.036373779 +0000 UTC m=+3842.469137672" watchObservedRunningTime="2026-10-30 08:01:22.037547759 +0000 UTC m=+3842.470311683"
Oct 30 08:01:22.381165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:22.381129    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:23.004345 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.004278    6738 generic.go:356] "Generic (PLEG): container finished" podID="bd3905c7-e765-4bbe-8422-80c740069251" containerID="b07d910d98f3d514e642866663c6386b7a7701ac30f4a01ffee00f1da59cad61" exitCode=0
Oct 30 08:01:23.004760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.004340    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb" event={"ID":"bd3905c7-e765-4bbe-8422-80c740069251","Type":"ContainerDied","Data":"b07d910d98f3d514e642866663c6386b7a7701ac30f4a01ffee00f1da59cad61"}
Oct 30 08:01:23.551231 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.551159    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 08:01:23.649076 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.649037    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\") pod \"19ae4571-bd10-4a89-860f-65de96573cf6\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") "
Oct 30 08:01:23.649270 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.649140    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\" (UniqueName: \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\") pod \"19ae4571-bd10-4a89-860f-65de96573cf6\" (UID: \"19ae4571-bd10-4a89-860f-65de96573cf6\") "
Oct 30 08:01:23.649557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.649520    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume" pod "19ae4571-bd10-4a89-860f-65de96573cf6" (UID: "19ae4571-bd10-4a89-860f-65de96573cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:23.651384 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.651359    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp" pod "19ae4571-bd10-4a89-860f-65de96573cf6" (UID: "19ae4571-bd10-4a89-860f-65de96573cf6"). InnerVolumeSpecName "kube-api-access-6dsbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:23.750461 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.750426    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ae4571-bd10-4a89-860f-65de96573cf6-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:23.750461 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:23.750456    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsbp\" (UniqueName: \"kubernetes.io/projected/19ae4571-bd10-4a89-860f-65de96573cf6-kube-api-access-6dsbp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:24.015210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.015174    6738 generic.go:356] "Generic (PLEG): container finished" podID="19ae4571-bd10-4a89-860f-65de96573cf6" containerID="baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537" exitCode=0
Oct 30 08:01:24.015716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.015219    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"
Oct 30 08:01:24.015716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.015256    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" event={"ID":"19ae4571-bd10-4a89-860f-65de96573cf6","Type":"ContainerDied","Data":"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"}
Oct 30 08:01:24.015716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.015323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v" event={"ID":"19ae4571-bd10-4a89-860f-65de96573cf6","Type":"ContainerDied","Data":"7132eba7b6d0a5c586a7efb5445df62ff5c272b94efa03bed408a6ecfc2cd29e"}
Oct 30 08:01:24.015716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.015350    6738 scope.go:122] "RemoveContainer" containerID="baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"
Oct 30 08:01:24.026508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.026467    6738 scope.go:122] "RemoveContainer" containerID="baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"
Oct 30 08:01:24.026829 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:24.026808    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537\": container with ID starting with baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537 not found: ID does not exist" containerID="baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"
Oct 30 08:01:24.026912 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.026837    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537"} err="failed to get container status \"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537\": rpc error: code = NotFound desc = could not find container \"baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537\": container with ID starting with baaab09cd7e30aaa00b5bc8335a66bc4054d282a7aa2a367fa12d6ef0c492537 not found: ID does not exist"
Oct 30 08:01:24.069723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.069692    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 08:01:24.085920 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.085880    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8e7948-767947c599-dx68v"]
Oct 30 08:01:24.145065 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.145023    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:24.182791 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.182766    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:24.359202 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.359109    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\" (UniqueName: \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\") pod \"bd3905c7-e765-4bbe-8422-80c740069251\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") "
Oct 30 08:01:24.359202 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.359154    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\") pod \"bd3905c7-e765-4bbe-8422-80c740069251\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") "
Oct 30 08:01:24.359521 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.359272    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\") pod \"bd3905c7-e765-4bbe-8422-80c740069251\" (UID: \"bd3905c7-e765-4bbe-8422-80c740069251\") "
Oct 30 08:01:24.359644 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.359622    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca" pod "bd3905c7-e765-4bbe-8422-80c740069251" (UID: "bd3905c7-e765-4bbe-8422-80c740069251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:24.359866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.359845    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:24.361513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.361493    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca" pod "bd3905c7-e765-4bbe-8422-80c740069251" (UID: "bd3905c7-e765-4bbe-8422-80c740069251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:24.361586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.361550    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq" pod "bd3905c7-e765-4bbe-8422-80c740069251" (UID: "bd3905c7-e765-4bbe-8422-80c740069251"). InnerVolumeSpecName "kube-api-access-dpqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:24.461394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.461351    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpqqq\" (UniqueName: \"kubernetes.io/projected/bd3905c7-e765-4bbe-8422-80c740069251-kube-api-access-dpqqq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:24.461394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:24.461387    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd3905c7-e765-4bbe-8422-80c740069251-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:25.026421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:25.026379    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb" event={"ID":"bd3905c7-e765-4bbe-8422-80c740069251","Type":"ContainerDied","Data":"3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860"}
Oct 30 08:01:25.026421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:25.026403    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"
Oct 30 08:01:25.026889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:25.026437    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd58a4ab9f7b7e81a128b6fe620f8d23a63d42b0e2937bac081df82b7c00860"
Oct 30 08:01:25.690569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:25.690534    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" path="/var/lib/kubelet/pods/19ae4571-bd10-4a89-860f-65de96573cf6/volumes"
Oct 30 08:01:25.939211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:25.939174    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:27.355638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.355596    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 08:01:27.356123 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.355847    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" podUID="8a9393ee-d855-4c96-83de-b2d039f93356" containerName="maas-api" containerID="cri-o://dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41" gracePeriod=30
Oct 30 08:01:27.472003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.471970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:27.617610 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.617538    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 08:01:27.857317 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.857259    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\") pod \"8a9393ee-d855-4c96-83de-b2d039f93356\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") "
Oct 30 08:01:27.857508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.857409    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\") pod \"8a9393ee-d855-4c96-83de-b2d039f93356\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") "
Oct 30 08:01:27.857599 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.857559    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\" (UniqueName: \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\") pod \"8a9393ee-d855-4c96-83de-b2d039f93356\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") "
Oct 30 08:01:27.861313 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.857688    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\") pod \"8a9393ee-d855-4c96-83de-b2d039f93356\" (UID: \"8a9393ee-d855-4c96-83de-b2d039f93356\") "
Oct 30 08:01:27.872377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.867768    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca" pod "8a9393ee-d855-4c96-83de-b2d039f93356" (UID: "8a9393ee-d855-4c96-83de-b2d039f93356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:27.906627 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.906579    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls" pod "8a9393ee-d855-4c96-83de-b2d039f93356" (UID: "8a9393ee-d855-4c96-83de-b2d039f93356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:27.918488 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.918450    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs" pod "8a9393ee-d855-4c96-83de-b2d039f93356" (UID: "8a9393ee-d855-4c96-83de-b2d039f93356"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:27.918875 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.918849    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj" pod "8a9393ee-d855-4c96-83de-b2d039f93356" (UID: "8a9393ee-d855-4c96-83de-b2d039f93356"). InnerVolumeSpecName "kube-api-access-8b6zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:27.961036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.961003    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:27.961254 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.961243    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9393ee-d855-4c96-83de-b2d039f93356-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:27.961359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.961347    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9393ee-d855-4c96-83de-b2d039f93356-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:27.961420 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:27.961412    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6zj\" (UniqueName: \"kubernetes.io/projected/8a9393ee-d855-4c96-83de-b2d039f93356-kube-api-access-8b6zj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:28.062141 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.062099    6738 generic.go:356] "Generic (PLEG): container finished" podID="8a9393ee-d855-4c96-83de-b2d039f93356" containerID="dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41" exitCode=0
Oct 30 08:01:28.062372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.062164    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"
Oct 30 08:01:28.062372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.062165    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" event={"ID":"8a9393ee-d855-4c96-83de-b2d039f93356","Type":"ContainerDied","Data":"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"}
Oct 30 08:01:28.062372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.062228    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr" event={"ID":"8a9393ee-d855-4c96-83de-b2d039f93356","Type":"ContainerDied","Data":"97df71fc46fb7340007b75b51da06efc369be3b4527405d9b28b9abce80bfb37"}
Oct 30 08:01:28.062372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.062257    6738 scope.go:122] "RemoveContainer" containerID="dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"
Oct 30 08:01:28.074323 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.074284    6738 scope.go:122] "RemoveContainer" containerID="dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"
Oct 30 08:01:28.074626 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:28.074605    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41\": container with ID starting with dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41 not found: ID does not exist" containerID="dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"
Oct 30 08:01:28.074670 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.074635    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41"} err="failed to get container status \"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41\": rpc error: code = NotFound desc = could not find container \"dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41\": container with ID starting with dcd2f6cbd40d22b347d95ea732567d4619a0a98201425c7798d818e68f996a41 not found: ID does not exist"
Oct 30 08:01:28.122376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.122340    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 08:01:28.127182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.127105    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-083c13ed-64c599dfcc-t5ndr"]
Oct 30 08:01:28.157017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.156974    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:28.274189 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.274153    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"]
Oct 30 08:01:28.277443 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.277418    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-083c13ed-mkx7z"]
Oct 30 08:01:28.342319 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.342226    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:28.460469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.460431    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"]
Oct 30 08:01:28.461114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461098    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8a9393ee-d855-4c96-83de-b2d039f93356" containerName="maas-api"
Oct 30 08:01:28.461165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461115    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9393ee-d855-4c96-83de-b2d039f93356" containerName="maas-api"
Oct 30 08:01:28.461165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461134    6738 cpu_manager.go:414] "containerMap: removing container" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing"
Oct 30 08:01:28.461165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461141    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing"
Oct 30 08:01:28.461165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461153    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8a431fec-7a10-4470-861a-ccb6092e6fd4" containerName="revoke-keys"
Oct 30 08:01:28.461165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461158    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a431fec-7a10-4470-861a-ccb6092e6fd4" containerName="revoke-keys"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461177    6738 cpu_manager.go:414] "containerMap: removing container" podUID="bd3905c7-e765-4bbe-8422-80c740069251" containerName="revoke-keys"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461182    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3905c7-e765-4bbe-8422-80c740069251" containerName="revoke-keys"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461288    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3905c7-e765-4bbe-8422-80c740069251" containerName="revoke-keys"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461317    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9393ee-d855-4c96-83de-b2d039f93356" containerName="maas-api"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461325    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ae4571-bd10-4a89-860f-65de96573cf6" containerName="payload-pre-processing"
Oct 30 08:01:28.461377 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.461337    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a431fec-7a10-4470-861a-ccb6092e6fd4" containerName="revoke-keys"
Oct 30 08:01:28.466441 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.466423    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.483905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.483871    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:28.483905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.483893    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:28.484927 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.484908    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"]
Oct 30 08:01:28.568727 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.568685    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.568944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.568822    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.568944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.568884    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5tq\" (UniqueName: \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.670023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.669926    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.670023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.670024    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5tq\" (UniqueName: \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.670282 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.670149    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.670672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.670650    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.672572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.672548    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.700033 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.699996    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5tq\" (UniqueName: \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\") pod \"maas-api-revoke-sub-86c01c38-r47hc\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.784998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.784968    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:28.936909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:28.936819    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:29.045717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.045683    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"]
Oct 30 08:01:29.050223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.050191    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfe54-qhd5h"]
Oct 30 08:01:29.092934 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.092901    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"]
Oct 30 08:01:29.093625 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:29.093594    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ddda41_f65b_4b9a_a9b7_4bd2d3f2e1d0.slice/crio-3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016 WatchSource:0}: Error finding container 3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016: Status 404 returned error can't find the container with id 3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016
Oct 30 08:01:29.692151 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.692119    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d7c427-c0a1-4e19-aaeb-5a255674871c" path="/var/lib/kubelet/pods/60d7c427-c0a1-4e19-aaeb-5a255674871c/volumes"
Oct 30 08:01:29.692627 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.692613    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9393ee-d855-4c96-83de-b2d039f93356" path="/var/lib/kubelet/pods/8a9393ee-d855-4c96-83de-b2d039f93356/volumes"
Oct 30 08:01:29.693040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:29.693027    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3e5173-b005-4a96-b7d3-354e24e4e945" path="/var/lib/kubelet/pods/da3e5173-b005-4a96-b7d3-354e24e4e945/volumes"
Oct 30 08:01:30.081718 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:30.081677    6738 generic.go:356] "Generic (PLEG): container finished" podID="48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" containerID="cba6fa2fc7fa746688b987676f4c33189228b2fda8fc9f353f178878f2680a25" exitCode=0
Oct 30 08:01:30.081941 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:30.081745    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc" event={"ID":"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0","Type":"ContainerDied","Data":"cba6fa2fc7fa746688b987676f4c33189228b2fda8fc9f353f178878f2680a25"}
Oct 30 08:01:30.081941 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:30.081796    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc" event={"ID":"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0","Type":"ContainerStarted","Data":"3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016"}
Oct 30 08:01:30.530569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:30.530534    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:31.166454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.166416    6738 scope.go:122] "RemoveContainer" containerID="95177c693fe43d5ad617ac45df85d44e4a406109619eab9dc2095838b6274d48"
Oct 30 08:01:31.252693 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.252663    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:31.308686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.308643    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\") pod \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") "
Oct 30 08:01:31.308915 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.308714    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\") pod \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") "
Oct 30 08:01:31.308915 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.308884    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\" (UniqueName: \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\") pod \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\" (UID: \"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0\") "
Oct 30 08:01:31.310571 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.310477    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca" pod "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" (UID: "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:31.312241 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.311524    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq" pod "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" (UID: "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0"). InnerVolumeSpecName "kube-api-access-kj5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:31.312780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.312726    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca" pod "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" (UID: "48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:31.411246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.411208    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:31.411246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.411241    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:31.411246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:31.411254    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5tq\" (UniqueName: \"kubernetes.io/projected/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0-kube-api-access-kj5tq\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:32.012619 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:32.012518    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:32.119889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:32.119843    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"
Oct 30 08:01:32.120082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:32.119905    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc" event={"ID":"48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0","Type":"ContainerDied","Data":"3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016"}
Oct 30 08:01:32.120082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:32.119963    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3359af667b258a90220b13b7d8598ae63dc57a3943456013285b199b1ef5a016"
Oct 30 08:01:32.351415 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:32.351286    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:34.242701 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:34.242667    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:35.760685 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:35.760634    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:35.943718 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:35.943676    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:36.428497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.428457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:36.630790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.630750    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:36.757994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.757903    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"]
Oct 30 08:01:36.769026 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.768993    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a475d6-t4btg"]
Oct 30 08:01:36.801566 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.801523    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:36.987669 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:36.987260    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:37.275549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:37.275510    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:37.519837 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:37.519799    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:01:37.693992 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:37.693801    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1028d367-ac06-421d-8c48-58753013e976" path="/var/lib/kubelet/pods/1028d367-ac06-421d-8c48-58753013e976/volumes"
Oct 30 08:01:38.081405 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:38.072469    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"]
Oct 30 08:01:38.097227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:38.092006    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cdefef5-t94sd"]
Oct 30 08:01:39.166681 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.166647    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"]
Oct 30 08:01:39.167411 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.167394    6738 cpu_manager.go:414] "containerMap: removing container" podUID="48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" containerName="revoke-keys"
Oct 30 08:01:39.167411 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.167412    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" containerName="revoke-keys"
Oct 30 08:01:39.167597 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.167581    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" containerName="revoke-keys"
Oct 30 08:01:39.172174 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.172150    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.195389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.195361    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:39.200939 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.200908    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"]
Oct 30 08:01:39.246671 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.246637    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8gx\" (UniqueName: \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.246852 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.246677    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.246852 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.246776    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.295635 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.295598    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:39.317897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.317866    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:39.348235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.348204    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8gx\" (UniqueName: \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.348492 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.348243    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.348492 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.348286    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.349180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.349151    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.350760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.350737    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.374491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.374459    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8gx\" (UniqueName: \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\") pod \"maas-api-revoke-sub-cfc57b52-v5gpz\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.487426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.487338    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:39.692693 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.692658    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895de860-6819-4d00-9d65-bf321d2685cf" path="/var/lib/kubelet/pods/895de860-6819-4d00-9d65-bf321d2685cf/volumes"
Oct 30 08:01:39.806505 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:39.806470    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c31806_3f63_42d6_9898_02c8a5e2e14e.slice/crio-cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680 WatchSource:0}: Error finding container cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680: Status 404 returned error can't find the container with id cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680
Oct 30 08:01:39.808451 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:39.808419    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"]
Oct 30 08:01:40.186179 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:40.186144    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz" event={"ID":"82c31806-3f63-42d6-9898-02c8a5e2e14e","Type":"ContainerStarted","Data":"34830ba31475a8a344984a9946e465bde7ef0faeba53e9078132e56b8de67359"}
Oct 30 08:01:40.186634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:40.186198    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz" event={"ID":"82c31806-3f63-42d6-9898-02c8a5e2e14e","Type":"ContainerStarted","Data":"cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680"}
Oct 30 08:01:40.240283 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:40.240222    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz" podStartSLOduration=1.240207001 podStartE2EDuration="1.240207001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:40.238897188 +0000 UTC m=+3860.671661081" watchObservedRunningTime="2026-10-30 08:01:40.240207001 +0000 UTC m=+3860.672970894"
Oct 30 08:01:41.090875 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.090771    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"]
Oct 30 08:01:41.103534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.103500    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f193e-nq287"]
Oct 30 08:01:41.198924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.198888    6738 generic.go:356] "Generic (PLEG): container finished" podID="82c31806-3f63-42d6-9898-02c8a5e2e14e" containerID="34830ba31475a8a344984a9946e465bde7ef0faeba53e9078132e56b8de67359" exitCode=0
Oct 30 08:01:41.199382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.198969    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz" event={"ID":"82c31806-3f63-42d6-9898-02c8a5e2e14e","Type":"ContainerDied","Data":"34830ba31475a8a344984a9946e465bde7ef0faeba53e9078132e56b8de67359"}
Oct 30 08:01:41.471164 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.471127    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:41.690924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:41.690889    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62943fed-8d63-4332-9309-5e89eb45fd2b" path="/var/lib/kubelet/pods/62943fed-8d63-4332-9309-5e89eb45fd2b/volumes"
Oct 30 08:01:42.367913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.367884    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:42.495774 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.495684    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\") pod \"82c31806-3f63-42d6-9898-02c8a5e2e14e\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") "
Oct 30 08:01:42.495977 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.495792    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\") pod \"82c31806-3f63-42d6-9898-02c8a5e2e14e\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") "
Oct 30 08:01:42.495977 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.495816    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\" (UniqueName: \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\") pod \"82c31806-3f63-42d6-9898-02c8a5e2e14e\" (UID: \"82c31806-3f63-42d6-9898-02c8a5e2e14e\") "
Oct 30 08:01:42.496133 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.496076    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca" pod "82c31806-3f63-42d6-9898-02c8a5e2e14e" (UID: "82c31806-3f63-42d6-9898-02c8a5e2e14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:42.496317 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.496284    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.498002 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.497973    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca" pod "82c31806-3f63-42d6-9898-02c8a5e2e14e" (UID: "82c31806-3f63-42d6-9898-02c8a5e2e14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:42.498194 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.498171    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx" pod "82c31806-3f63-42d6-9898-02c8a5e2e14e" (UID: "82c31806-3f63-42d6-9898-02c8a5e2e14e"). InnerVolumeSpecName "kube-api-access-8s8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:42.597952 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.597914    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c31806-3f63-42d6-9898-02c8a5e2e14e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.597952 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:42.597946    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8gx\" (UniqueName: \"kubernetes.io/projected/82c31806-3f63-42d6-9898-02c8a5e2e14e-kube-api-access-8s8gx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:43.217213 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:43.217169    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz" event={"ID":"82c31806-3f63-42d6-9898-02c8a5e2e14e","Type":"ContainerDied","Data":"cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680"}
Oct 30 08:01:43.217460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:43.217236    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd488c9546f166f003fdccfd25b7609853d5de570fd3bb9051e77b0876dae680"
Oct 30 08:01:43.217460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:43.217189    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"
Oct 30 08:01:43.238507 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:43.238473    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:44.495922 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:44.495886    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:44.631886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:44.631850    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:44.942216 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:44.942181    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:45.093925 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.093889    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:45.437012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.436971    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"]
Oct 30 08:01:45.437800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.437778    6738 cpu_manager.go:414] "containerMap: removing container" podUID="82c31806-3f63-42d6-9898-02c8a5e2e14e" containerName="revoke-keys"
Oct 30 08:01:45.437800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.437799    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c31806-3f63-42d6-9898-02c8a5e2e14e" containerName="revoke-keys"
Oct 30 08:01:45.437965 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.437914    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c31806-3f63-42d6-9898-02c8a5e2e14e" containerName="revoke-keys"
Oct 30 08:01:45.442284 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.442265    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.462068 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.462032    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"]
Oct 30 08:01:45.469171 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.469142    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:45.469402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.469208    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:45.540923 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.540880    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.541389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.541043    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vksw\" (UniqueName: \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.541389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.541153    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.642509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.642462    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.642736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.642605    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.642736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.642703    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vksw\" (UniqueName: \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.643285 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.643264    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.645011 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.644986    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.665873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.665839    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vksw\" (UniqueName: \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:45.756864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:45.756778    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:46.072195 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:46.072159    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204dad49_5512_4e70_99f2_b6bb0657eb5e.slice/crio-ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7 WatchSource:0}: Error finding container ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7: Status 404 returned error can't find the container with id ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7
Oct 30 08:01:46.076206 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.076172    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"]
Oct 30 08:01:46.244350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.244305    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl" event={"ID":"204dad49-5512-4e70-99f2-b6bb0657eb5e","Type":"ContainerStarted","Data":"ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7"}
Oct 30 08:01:46.330980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.330887    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:46.602271 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.602167    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"]
Oct 30 08:01:46.607672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.607638    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.611778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.611745    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"]
Oct 30 08:01:46.758840 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.758795    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cd4c\" (UniqueName: \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.759079 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.758968    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.759168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.759080    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.860394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.860264    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cd4c\" (UniqueName: \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.860586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.860396    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.860586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.860476    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.861088 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.861066    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.862816 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.862799    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.884037 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.884004    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cd4c\" (UniqueName: \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:46.925030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:46.924992    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:47.187195 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:47.187160    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:47.276673 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:47.276154    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"]
Oct 30 08:01:47.292337 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:47.292282    6738 generic.go:356] "Generic (PLEG): container finished" podID="204dad49-5512-4e70-99f2-b6bb0657eb5e" containerID="74ecc93acfcc886a66a3b3d045b784cd562af70eeb5dfd24af343b2bb88fc118" exitCode=0
Oct 30 08:01:47.292541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:47.292467    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl" event={"ID":"204dad49-5512-4e70-99f2-b6bb0657eb5e","Type":"ContainerDied","Data":"74ecc93acfcc886a66a3b3d045b784cd562af70eeb5dfd24af343b2bb88fc118"}
Oct 30 08:01:48.302771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.302732    6738 generic.go:356] "Generic (PLEG): container finished" podID="9bc0a030-5530-457b-baed-e384bbbd1ae7" containerID="36a6e4a18d29f0a024edb3e64c3c52e7f135954fdfc3c2f8a5d15e029aa1f3d8" exitCode=0
Oct 30 08:01:48.303255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.302815    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm" event={"ID":"9bc0a030-5530-457b-baed-e384bbbd1ae7","Type":"ContainerDied","Data":"36a6e4a18d29f0a024edb3e64c3c52e7f135954fdfc3c2f8a5d15e029aa1f3d8"}
Oct 30 08:01:48.303255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.302871    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm" event={"ID":"9bc0a030-5530-457b-baed-e384bbbd1ae7","Type":"ContainerStarted","Data":"f33fd3cfc7ed38345c6c3630e2e1103f32491cfe246d47b4370b919c24560684"}
Oct 30 08:01:48.461620 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.461595    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:48.584329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.584261    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\") pod \"204dad49-5512-4e70-99f2-b6bb0657eb5e\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") "
Oct 30 08:01:48.584540 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.584378    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\") pod \"204dad49-5512-4e70-99f2-b6bb0657eb5e\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") "
Oct 30 08:01:48.584540 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.584521    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\" (UniqueName: \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\") pod \"204dad49-5512-4e70-99f2-b6bb0657eb5e\" (UID: \"204dad49-5512-4e70-99f2-b6bb0657eb5e\") "
Oct 30 08:01:48.584765 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.584738    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca" pod "204dad49-5512-4e70-99f2-b6bb0657eb5e" (UID: "204dad49-5512-4e70-99f2-b6bb0657eb5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:48.585012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.584994    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:48.586516 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.586492    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca" pod "204dad49-5512-4e70-99f2-b6bb0657eb5e" (UID: "204dad49-5512-4e70-99f2-b6bb0657eb5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:48.586678 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.586659    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw" pod "204dad49-5512-4e70-99f2-b6bb0657eb5e" (UID: "204dad49-5512-4e70-99f2-b6bb0657eb5e"). InnerVolumeSpecName "kube-api-access-2vksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:48.686330 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.686249    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204dad49-5512-4e70-99f2-b6bb0657eb5e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:48.686330 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:48.686281    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vksw\" (UniqueName: \"kubernetes.io/projected/204dad49-5512-4e70-99f2-b6bb0657eb5e-kube-api-access-2vksw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:49.112914 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.112825    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:49.312329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.312280    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"
Oct 30 08:01:49.312329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.312316    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl" event={"ID":"204dad49-5512-4e70-99f2-b6bb0657eb5e","Type":"ContainerDied","Data":"ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7"}
Oct 30 08:01:49.312806 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.312365    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac09196f7bd186f70a0b268e306459a1edf4f19691f49fb36080bfe3394b7fa7"
Oct 30 08:01:49.484771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.484741    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:49.604606 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.604574    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\") pod \"9bc0a030-5530-457b-baed-e384bbbd1ae7\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") "
Oct 30 08:01:49.604815 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.604621    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\") pod \"9bc0a030-5530-457b-baed-e384bbbd1ae7\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") "
Oct 30 08:01:49.604815 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.604654    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\" (UniqueName: \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\") pod \"9bc0a030-5530-457b-baed-e384bbbd1ae7\" (UID: \"9bc0a030-5530-457b-baed-e384bbbd1ae7\") "
Oct 30 08:01:49.605042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.605013    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca" pod "9bc0a030-5530-457b-baed-e384bbbd1ae7" (UID: "9bc0a030-5530-457b-baed-e384bbbd1ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:49.606808 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.606785    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca" pod "9bc0a030-5530-457b-baed-e384bbbd1ae7" (UID: "9bc0a030-5530-457b-baed-e384bbbd1ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:49.606884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.606826    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c" pod "9bc0a030-5530-457b-baed-e384bbbd1ae7" (UID: "9bc0a030-5530-457b-baed-e384bbbd1ae7"). InnerVolumeSpecName "kube-api-access-2cd4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:49.706173 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.706143    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:49.706359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.706177    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc0a030-5530-457b-baed-e384bbbd1ae7-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:49.706359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:49.706193    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cd4c\" (UniqueName: \"kubernetes.io/projected/9bc0a030-5530-457b-baed-e384bbbd1ae7-kube-api-access-2cd4c\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:01:49.708659 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:49.708631    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc0a030_5530_457b_baed_e384bbbd1ae7.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 08:01:49.708756 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:01:49.708651    6738 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc0a030_5530_457b_baed_e384bbbd1ae7.slice/crio-f33fd3cfc7ed38345c6c3630e2e1103f32491cfe246d47b4370b919c24560684\": RecentStats: unable to find data in memory cache]"
Oct 30 08:01:50.322357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:50.322323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm" event={"ID":"9bc0a030-5530-457b-baed-e384bbbd1ae7","Type":"ContainerDied","Data":"f33fd3cfc7ed38345c6c3630e2e1103f32491cfe246d47b4370b919c24560684"}
Oct 30 08:01:50.322754 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:50.322405    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f33fd3cfc7ed38345c6c3630e2e1103f32491cfe246d47b4370b919c24560684"
Oct 30 08:01:50.322754 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:50.322334    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"
Oct 30 08:01:50.660101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:50.660063    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:51.278753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.278711    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:51.408960 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.408924    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:51.422735 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.422694    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"]
Oct 30 08:01:51.449388 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.449352    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1f700-hlcqd"]
Oct 30 08:01:51.691268 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.691236    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af5f6ee-7f5f-4ec4-a326-9818a29a1911" path="/var/lib/kubelet/pods/8af5f6ee-7f5f-4ec4-a326-9818a29a1911/volumes"
Oct 30 08:01:51.908606 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:51.908572    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:53.347756 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:53.347710    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:55.225461 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:55.225430    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:55.962498 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:55.962449    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 08:01:55.997012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:55.996977    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 08:01:56.120051 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.120021    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:56.248383 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.239868    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:56.447946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.447913    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:01:56.448705 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.448683    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9bc0a030-5530-457b-baed-e384bbbd1ae7" containerName="revoke-keys"
Oct 30 08:01:56.448797 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.448708    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc0a030-5530-457b-baed-e384bbbd1ae7" containerName="revoke-keys"
Oct 30 08:01:56.448797 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.448782    6738 cpu_manager.go:414] "containerMap: removing container" podUID="204dad49-5512-4e70-99f2-b6bb0657eb5e" containerName="revoke-keys"
Oct 30 08:01:56.448797 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.448791    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204dad49-5512-4e70-99f2-b6bb0657eb5e" containerName="revoke-keys"
Oct 30 08:01:56.448984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.448966    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="204dad49-5512-4e70-99f2-b6bb0657eb5e" containerName="revoke-keys"
Oct 30 08:01:56.449036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.449001    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc0a030-5530-457b-baed-e384bbbd1ae7" containerName="revoke-keys"
Oct 30 08:01:56.452463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.452440    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.468892 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.468858    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"default-dockercfg-qtpw7\""
Oct 30 08:01:56.469090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.468858    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"rate-test-model-14271d-kserve-self-signed-certs\""
Oct 30 08:01:56.469090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.469032    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"openshift-service-ca.crt\""
Oct 30 08:01:56.469090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.469032    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"kube-root-ca.crt\""
Oct 30 08:01:56.469488 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.469465    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:01:56.526784 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.526695    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.526959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.526849    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhgw\" (UniqueName: \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.526959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.526886    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.527042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.526997    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.527042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.527024    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.527143 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.527123    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.544157 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.544110    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:56.628680 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628642    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.628891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628750    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhgw\" (UniqueName: \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.628891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628773    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.628891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628829    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.628891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628871    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.629100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.628945    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.629257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.629229    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.629367 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.629237    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.629443 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.629427    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.631184 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.631161    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.631285 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.631259    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.653308 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.653250    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhgw\" (UniqueName: \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\") pod \"rate-test-model-14271d-kserve-7ff4fc5597-7nb2g\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") " pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:56.773760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:56.773475    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:57.003064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.003002    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:57.178759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.178702    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:01:57.192954 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:01:57.192883    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ff2db4b_8b43_4210_a92c_de75c4b7a689.slice/crio-6873353b778af0a307918d265dc847bd80bccc5182da69f349d961a87329c0fa WatchSource:0}: Error finding container 6873353b778af0a307918d265dc847bd80bccc5182da69f349d961a87329c0fa: Status 404 returned error can't find the container with id 6873353b778af0a307918d265dc847bd80bccc5182da69f349d961a87329c0fa
Oct 30 08:01:57.380433 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.380393    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" event={"ID":"5ff2db4b-8b43-4210-a92c-de75c4b7a689","Type":"ContainerStarted","Data":"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"}
Oct 30 08:01:57.381131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.380439    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" event={"ID":"5ff2db4b-8b43-4210-a92c-de75c4b7a689","Type":"ContainerStarted","Data":"6873353b778af0a307918d265dc847bd80bccc5182da69f349d961a87329c0fa"}
Oct 30 08:01:57.423092 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.423034    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" podStartSLOduration=1.423019869 podStartE2EDuration="1.423019869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:57.421645199 +0000 UTC m=+3877.854409091" watchObservedRunningTime="2026-10-30 08:01:57.423019869 +0000 UTC m=+3877.855783762"
Oct 30 08:01:57.616369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.606321    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 08:01:57.662339 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.661950    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 08:01:57.768443 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.768398    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:57.782483 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.782433    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:01:57.910088 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:57.910000    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:58.666779 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:58.666743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:58.785518 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:58.785483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:58.905400 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:58.904156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:59.038834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.038707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:01:59.637571 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.637536    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"]
Oct 30 08:01:59.641811 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.641791    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.658818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.658790    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"]
Oct 30 08:01:59.740010 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.739973    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:01:59.740010 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.740006    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:01:59.743670 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.743644    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.743814 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.743720    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.744188 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.744115    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5pp\" (UniqueName: \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.845103 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.845062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5pp\" (UniqueName: \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.845344 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.845114    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.845344 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.845177    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.845962 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.845935    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.847739 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.847710    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.869178 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.869144    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5pp\" (UniqueName: \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:01:59.958455 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:01:59.958364    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:02:00.267880 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:00.267836    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf232b311_a183_42c3_bbbe_59e4a95525c3.slice/crio-fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1 WatchSource:0}: Error finding container fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1: Status 404 returned error can't find the container with id fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1
Oct 30 08:02:00.268024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:00.267873    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"]
Oct 30 08:02:00.414040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:00.413992    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55" event={"ID":"f232b311-a183-42c3-bbbe-59e4a95525c3","Type":"ContainerStarted","Data":"fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1"}
Oct 30 08:02:00.722400 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:00.722359    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:00.962799 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:00.962712    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" podUID="53090933-8414-4e71-aa9f-b79991927dc7" containerName="payload-processing" containerID="cri-o://c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813" gracePeriod=25
Oct 30 08:02:00.997322 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:00.997280    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" podUID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" containerName="payload-pre-processing" containerID="cri-o://ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf" gracePeriod=25
Oct 30 08:02:01.292873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.292833    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 08:02:01.310102 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.310069    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 08:02:01.368069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368034    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\") pod \"53090933-8414-4e71-aa9f-b79991927dc7\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") "
Oct 30 08:02:01.368311 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368179    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\") pod \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") "
Oct 30 08:02:01.368394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\" (UniqueName: \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\") pod \"53090933-8414-4e71-aa9f-b79991927dc7\" (UID: \"53090933-8414-4e71-aa9f-b79991927dc7\") "
Oct 30 08:02:01.368458 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368418    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\" (UniqueName: \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\") pod \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\" (UID: \"3cfaa571-f6ba-45da-bf89-b8448fa096b2\") "
Oct 30 08:02:01.368665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368620    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume" pod "3cfaa571-f6ba-45da-bf89-b8448fa096b2" (UID: "3cfaa571-f6ba-45da-bf89-b8448fa096b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:01.368782 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.368757    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume" pod "53090933-8414-4e71-aa9f-b79991927dc7" (UID: "53090933-8414-4e71-aa9f-b79991927dc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:01.369090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.369063    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfaa571-f6ba-45da-bf89-b8448fa096b2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.369262 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.369095    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53090933-8414-4e71-aa9f-b79991927dc7-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.370801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.370777    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5" pod "3cfaa571-f6ba-45da-bf89-b8448fa096b2" (UID: "3cfaa571-f6ba-45da-bf89-b8448fa096b2"). InnerVolumeSpecName "kube-api-access-gtkq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:01.370913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.370832    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6" pod "53090933-8414-4e71-aa9f-b79991927dc7" (UID: "53090933-8414-4e71-aa9f-b79991927dc7"). InnerVolumeSpecName "kube-api-access-hn4z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:01.425694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.425660    6738 generic.go:356] "Generic (PLEG): container finished" podID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" containerID="ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf" exitCode=0
Oct 30 08:02:01.425916 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.425703    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"
Oct 30 08:02:01.425916 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.425754    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" event={"ID":"3cfaa571-f6ba-45da-bf89-b8448fa096b2","Type":"ContainerDied","Data":"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"}
Oct 30 08:02:01.425916 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.425879    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9" event={"ID":"3cfaa571-f6ba-45da-bf89-b8448fa096b2","Type":"ContainerDied","Data":"98c68dedd2ec6d89701aa1c0e7f22017a81463b1b0a79ae08ca00e028d978959"}
Oct 30 08:02:01.425916 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.425897    6738 scope.go:122] "RemoveContainer" containerID="ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"
Oct 30 08:02:01.427878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.427852    6738 generic.go:356] "Generic (PLEG): container finished" podID="f232b311-a183-42c3-bbbe-59e4a95525c3" containerID="aaf3b90b840797595ca36db33e99b38a3aca95b00f93c2a30a78c8a234d08dea" exitCode=0
Oct 30 08:02:01.428031 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.427918    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55" event={"ID":"f232b311-a183-42c3-bbbe-59e4a95525c3","Type":"ContainerDied","Data":"aaf3b90b840797595ca36db33e99b38a3aca95b00f93c2a30a78c8a234d08dea"}
Oct 30 08:02:01.429633 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.429601    6738 generic.go:356] "Generic (PLEG): container finished" podID="53090933-8414-4e71-aa9f-b79991927dc7" containerID="c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813" exitCode=0
Oct 30 08:02:01.429780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.429659    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"
Oct 30 08:02:01.429780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.429693    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" event={"ID":"53090933-8414-4e71-aa9f-b79991927dc7","Type":"ContainerDied","Data":"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"}
Oct 30 08:02:01.429904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.429779    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf" event={"ID":"53090933-8414-4e71-aa9f-b79991927dc7","Type":"ContainerDied","Data":"12fcf4514be22f61d832d315b7a4fd3e0a75303624bbb5c51c8faa98a3e1cb2e"}
Oct 30 08:02:01.438496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.438476    6738 scope.go:122] "RemoveContainer" containerID="ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"
Oct 30 08:02:01.438844 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:01.438817    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf\": container with ID starting with ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf not found: ID does not exist" containerID="ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"
Oct 30 08:02:01.438932 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.438856    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf"} err="failed to get container status \"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf\": rpc error: code = NotFound desc = could not find container \"ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf\": container with ID starting with ecdb78112905000090d846eeca7dfbfafd9f91a0852078935a51b914aca8fddf not found: ID does not exist"
Oct 30 08:02:01.438932 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.438879    6738 scope.go:122] "RemoveContainer" containerID="c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"
Oct 30 08:02:01.450950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.450931    6738 scope.go:122] "RemoveContainer" containerID="c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"
Oct 30 08:02:01.451273 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:01.451250    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813\": container with ID starting with c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813 not found: ID does not exist" containerID="c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"
Oct 30 08:02:01.451372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.451282    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813"} err="failed to get container status \"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813\": rpc error: code = NotFound desc = could not find container \"c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813\": container with ID starting with c72045dbf8da835125c017e8cd215c9d32e1fb3e64fbece2e6c685eef7f07813 not found: ID does not exist"
Oct 30 08:02:01.469980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.469947    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4z6\" (UniqueName: \"kubernetes.io/projected/53090933-8414-4e71-aa9f-b79991927dc7-kube-api-access-hn4z6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.469980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.469976    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkq5\" (UniqueName: \"kubernetes.io/projected/3cfaa571-f6ba-45da-bf89-b8448fa096b2-kube-api-access-gtkq5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.482969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.482491    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 08:02:01.486175 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.486144    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d09eb6-7946db469d-gzmkf"]
Oct 30 08:02:01.524236 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.524147    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 08:02:01.529475 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.529443    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d09eb6-769d4bb49f-r6cs9"]
Oct 30 08:02:01.691462 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.691432    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" path="/var/lib/kubelet/pods/3cfaa571-f6ba-45da-bf89-b8448fa096b2/volumes"
Oct 30 08:02:01.691883 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:01.691870    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53090933-8414-4e71-aa9f-b79991927dc7" path="/var/lib/kubelet/pods/53090933-8414-4e71-aa9f-b79991927dc7/volumes"
Oct 30 08:02:02.670331 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.669835    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing" containerID="cri-o://b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0" gracePeriod=25
Oct 30 08:02:02.670902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.670468    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing" containerID="cri-o://0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df" gracePeriod=25
Oct 30 08:02:02.761726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.761689    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:02:02.778085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.778037    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:02.864205 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.864158    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\") pod \"f232b311-a183-42c3-bbbe-59e4a95525c3\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") "
Oct 30 08:02:02.864489 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.864348    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\") pod \"f232b311-a183-42c3-bbbe-59e4a95525c3\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") "
Oct 30 08:02:02.864566 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.864529    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\" (UniqueName: \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\") pod \"f232b311-a183-42c3-bbbe-59e4a95525c3\" (UID: \"f232b311-a183-42c3-bbbe-59e4a95525c3\") "
Oct 30 08:02:02.907321 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.906181    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca" pod "f232b311-a183-42c3-bbbe-59e4a95525c3" (UID: "f232b311-a183-42c3-bbbe-59e4a95525c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:02.909390 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.909346    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca" pod "f232b311-a183-42c3-bbbe-59e4a95525c3" (UID: "f232b311-a183-42c3-bbbe-59e4a95525c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:02.927643 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.927566    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp" pod "f232b311-a183-42c3-bbbe-59e4a95525c3" (UID: "f232b311-a183-42c3-bbbe-59e4a95525c3"). InnerVolumeSpecName "kube-api-access-4j5pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:02.966453 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.966419    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5pp\" (UniqueName: \"kubernetes.io/projected/f232b311-a183-42c3-bbbe-59e4a95525c3-kube-api-access-4j5pp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:02.966453 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.966450    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:02.966453 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:02.966460    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f232b311-a183-42c3-bbbe-59e4a95525c3-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:03.485714 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:03.485683    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"
Oct 30 08:02:03.485714 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:03.485703    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55" event={"ID":"f232b311-a183-42c3-bbbe-59e4a95525c3","Type":"ContainerDied","Data":"fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1"}
Oct 30 08:02:03.485972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:03.485761    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc89dbce0a6211e97730442a6a66b199e09dc3118e5c7e84377c8b655fd841d1"
Oct 30 08:02:04.596644 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:04.596608    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:06.268985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:06.268942    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:06.268985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:06.268984    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:06.342687 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:06.342645    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:06.775163 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:06.775128    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:02:06.788448 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:06.788418    6738 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:02:07.520541 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:07.520504    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:02:07.539353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:07.539324    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:02:07.867498 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:07.867391    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:08.563461 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:08.563426    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:08.676366 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:08.676329    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:08.694834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:08.694790    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"]
Oct 30 08:02:08.707409 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:08.707373    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bec5c-fb7fm"]
Oct 30 08:02:08.795314 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:08.795060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:09.240061 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:09.240018    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:09.692482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:09.692449    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc0a030-5530-457b-baed-e384bbbd1ae7" path="/var/lib/kubelet/pods/9bc0a030-5530-457b-baed-e384bbbd1ae7/volumes"
Oct 30 08:02:09.847246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:09.847210    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 08:02:09.879787 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:09.879750    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 08:02:09.995070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:09.994966    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:10.117430 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:10.117389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:10.707547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:10.704449    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:12.338180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:12.338142    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:14.032626 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:14.032592    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:14.847718 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:14.847670    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" podUID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" containerName="payload-processing" containerID="cri-o://730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb" gracePeriod=25
Oct 30 08:02:14.876709 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:14.876676    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:14.880349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:14.880321    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" podUID="359e565c-47a4-4762-9da7-00c983cb06b2" containerName="payload-pre-processing" containerID="cri-o://efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710" gracePeriod=25
Oct 30 08:02:15.146019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.145988    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 08:02:15.187610 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.187580    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 08:02:15.222242 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222201    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\" (UniqueName: \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\") pod \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") "
Oct 30 08:02:15.222486 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222449    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\" (UniqueName: \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\") pod \"359e565c-47a4-4762-9da7-00c983cb06b2\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") "
Oct 30 08:02:15.222575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222491    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\") pod \"359e565c-47a4-4762-9da7-00c983cb06b2\" (UID: \"359e565c-47a4-4762-9da7-00c983cb06b2\") "
Oct 30 08:02:15.222575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222538    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\") pod \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\" (UID: \"85041ce5-3e68-4edc-b5d2-9e0a3e880a83\") "
Oct 30 08:02:15.222932 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222908    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume" pod "359e565c-47a4-4762-9da7-00c983cb06b2" (UID: "359e565c-47a4-4762-9da7-00c983cb06b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:15.223033 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.222995    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume" pod "85041ce5-3e68-4edc-b5d2-9e0a3e880a83" (UID: "85041ce5-3e68-4edc-b5d2-9e0a3e880a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:15.223161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.223142    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359e565c-47a4-4762-9da7-00c983cb06b2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:15.223218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.223168    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:15.224959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.224931    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc" pod "85041ce5-3e68-4edc-b5d2-9e0a3e880a83" (UID: "85041ce5-3e68-4edc-b5d2-9e0a3e880a83"). InnerVolumeSpecName "kube-api-access-k7ptc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:15.225074 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.225050    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd" pod "359e565c-47a4-4762-9da7-00c983cb06b2" (UID: "359e565c-47a4-4762-9da7-00c983cb06b2"). InnerVolumeSpecName "kube-api-access-6mdwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:15.324744 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.324692    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdwd\" (UniqueName: \"kubernetes.io/projected/359e565c-47a4-4762-9da7-00c983cb06b2-kube-api-access-6mdwd\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:15.324744 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.324742    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7ptc\" (UniqueName: \"kubernetes.io/projected/85041ce5-3e68-4edc-b5d2-9e0a3e880a83-kube-api-access-k7ptc\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:15.677004 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.676967    6738 generic.go:356] "Generic (PLEG): container finished" podID="359e565c-47a4-4762-9da7-00c983cb06b2" containerID="efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710" exitCode=0
Oct 30 08:02:15.677198 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.677020    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"
Oct 30 08:02:15.677198 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.677043    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" event={"ID":"359e565c-47a4-4762-9da7-00c983cb06b2","Type":"ContainerDied","Data":"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"}
Oct 30 08:02:15.677198 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.677122    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p" event={"ID":"359e565c-47a4-4762-9da7-00c983cb06b2","Type":"ContainerDied","Data":"0ac94ef8d0baa27dd451cca96c0c7053b19617e534f3dab6846da70ad978d862"}
Oct 30 08:02:15.677198 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.677166    6738 scope.go:122] "RemoveContainer" containerID="efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"
Oct 30 08:02:15.678983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.678819    6738 generic.go:356] "Generic (PLEG): container finished" podID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" containerID="730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb" exitCode=0
Oct 30 08:02:15.678983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.678879    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"
Oct 30 08:02:15.678983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.678896    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" event={"ID":"85041ce5-3e68-4edc-b5d2-9e0a3e880a83","Type":"ContainerDied","Data":"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"}
Oct 30 08:02:15.678983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.678942    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv" event={"ID":"85041ce5-3e68-4edc-b5d2-9e0a3e880a83","Type":"ContainerDied","Data":"f550d5c88132fac8512e4f2e8ed6e41aeef18733c9f6b04bde9a6b294d035f2b"}
Oct 30 08:02:15.691876 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.691850    6738 scope.go:122] "RemoveContainer" containerID="efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"
Oct 30 08:02:15.692237 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:15.692213    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710\": container with ID starting with efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710 not found: ID does not exist" containerID="efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"
Oct 30 08:02:15.692370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.692247    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710"} err="failed to get container status \"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710\": rpc error: code = NotFound desc = could not find container \"efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710\": container with ID starting with efdac9681cfe9b488ea415ed970bd1a1afe75c378950738b265a61b9900cc710 not found: ID does not exist"
Oct 30 08:02:15.692370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.692265    6738 scope.go:122] "RemoveContainer" containerID="730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"
Oct 30 08:02:15.704546 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.704502    6738 scope.go:122] "RemoveContainer" containerID="730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"
Oct 30 08:02:15.704905 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:15.704881    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb\": container with ID starting with 730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb not found: ID does not exist" containerID="730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"
Oct 30 08:02:15.704973 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.704914    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb"} err="failed to get container status \"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb\": rpc error: code = NotFound desc = could not find container \"730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb\": container with ID starting with 730e4db746dc5ccbf247b0c404adfcf2a667231503dde9ca2a8f4bbe311c7acb not found: ID does not exist"
Oct 30 08:02:15.731591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.731552    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 08:02:15.743432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.743402    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-265384-7c4d955b59-ngp6p"]
Oct 30 08:02:15.780142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.780106    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 08:02:15.785493 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.785464    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-265384-6b77d4c4b6-z5plv"]
Oct 30 08:02:15.813463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:15.813427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:16.268252 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:16.268212    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:16.268711 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:16.268252    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:16.720972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:16.720938    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:16.939315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:16.935959    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:17.062601 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.062520    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:17.587260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.587220    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:17.692111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.692071    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359e565c-47a4-4762-9da7-00c983cb06b2" path="/var/lib/kubelet/pods/359e565c-47a4-4762-9da7-00c983cb06b2/volumes"
Oct 30 08:02:17.692581 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.692567    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" path="/var/lib/kubelet/pods/85041ce5-3e68-4edc-b5d2-9e0a3e880a83/volumes"
Oct 30 08:02:17.762978 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.762940    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:17.931911 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.931874    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"]
Oct 30 08:02:17.932600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932580    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" containerName="payload-pre-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932601    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" containerName="payload-pre-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932627    6738 cpu_manager.go:414] "containerMap: removing container" podUID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" containerName="payload-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932635    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" containerName="payload-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932681    6738 cpu_manager.go:414] "containerMap: removing container" podUID="53090933-8414-4e71-aa9f-b79991927dc7" containerName="payload-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932689    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53090933-8414-4e71-aa9f-b79991927dc7" containerName="payload-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932704    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f232b311-a183-42c3-bbbe-59e4a95525c3" containerName="revoke-keys"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932711    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f232b311-a183-42c3-bbbe-59e4a95525c3" containerName="revoke-keys"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932731    6738 cpu_manager.go:414] "containerMap: removing container" podUID="359e565c-47a4-4762-9da7-00c983cb06b2" containerName="payload-pre-processing"
Oct 30 08:02:17.932755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932739    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359e565c-47a4-4762-9da7-00c983cb06b2" containerName="payload-pre-processing"
Oct 30 08:02:17.933257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932918    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="53090933-8414-4e71-aa9f-b79991927dc7" containerName="payload-processing"
Oct 30 08:02:17.933257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932933    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f232b311-a183-42c3-bbbe-59e4a95525c3" containerName="revoke-keys"
Oct 30 08:02:17.933257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932942    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="359e565c-47a4-4762-9da7-00c983cb06b2" containerName="payload-pre-processing"
Oct 30 08:02:17.933257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932962    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfaa571-f6ba-45da-bf89-b8448fa096b2" containerName="payload-pre-processing"
Oct 30 08:02:17.933257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.932974    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="85041ce5-3e68-4edc-b5d2-9e0a3e880a83" containerName="payload-processing"
Oct 30 08:02:17.938449 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.938421    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:17.953715 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.953687    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"]
Oct 30 08:02:17.962040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.962012    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:02:17.962236 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:17.962129    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:02:18.075263 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.075224    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.075481 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.075406    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzkx\" (UniqueName: \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.075481 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.075449    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.176749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.176702    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.176968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.176797    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzkx\" (UniqueName: \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.176968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.176827    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.177487 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.177462    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.179361 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.179337    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.200882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.200811    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzkx\" (UniqueName: \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\") pod \"maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.253998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.253962    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:18.564128 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:18.564093    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfc3fe3_2201_4b00_9342_621f4d96e281.slice/crio-1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51 WatchSource:0}: Error finding container 1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51: Status 404 returned error can't find the container with id 1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51
Oct 30 08:02:18.564606 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.564582    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"]
Oct 30 08:02:18.707752 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:18.707714    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc" event={"ID":"ebfc3fe3-2201-4b00-9342-621f4d96e281","Type":"ContainerStarted","Data":"1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51"}
Oct 30 08:02:19.718478 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:19.718400    6738 generic.go:356] "Generic (PLEG): container finished" podID="ebfc3fe3-2201-4b00-9342-621f4d96e281" containerID="2540d1986043989c12e8b32d68b786d9ebb14668102942cff4a12a4a8067cd65" exitCode=0
Oct 30 08:02:19.718869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:19.718477    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc" event={"ID":"ebfc3fe3-2201-4b00-9342-621f4d96e281","Type":"ContainerDied","Data":"2540d1986043989c12e8b32d68b786d9ebb14668102942cff4a12a4a8067cd65"}
Oct 30 08:02:19.761863 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:19.761829    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:20.904683 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:20.904654    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:21.030714 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.030623    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\") pod \"ebfc3fe3-2201-4b00-9342-621f4d96e281\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") "
Oct 30 08:02:21.030714 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.030680    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\") pod \"ebfc3fe3-2201-4b00-9342-621f4d96e281\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") "
Oct 30 08:02:21.030984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.030864    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\" (UniqueName: \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\") pod \"ebfc3fe3-2201-4b00-9342-621f4d96e281\" (UID: \"ebfc3fe3-2201-4b00-9342-621f4d96e281\") "
Oct 30 08:02:21.031129 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.031105    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca" pod "ebfc3fe3-2201-4b00-9342-621f4d96e281" (UID: "ebfc3fe3-2201-4b00-9342-621f4d96e281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:21.031284 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.031267    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:21.032867 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.032846    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca" pod "ebfc3fe3-2201-4b00-9342-621f4d96e281" (UID: "ebfc3fe3-2201-4b00-9342-621f4d96e281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:21.032974 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.032960    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx" pod "ebfc3fe3-2201-4b00-9342-621f4d96e281" (UID: "ebfc3fe3-2201-4b00-9342-621f4d96e281"). InnerVolumeSpecName "kube-api-access-9jzkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:21.132434 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.132394    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jzkx\" (UniqueName: \"kubernetes.io/projected/ebfc3fe3-2201-4b00-9342-621f4d96e281-kube-api-access-9jzkx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:21.132434 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.132426    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfc3fe3-2201-4b00-9342-621f4d96e281-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:21.745019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.744918    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:21.745616 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.745581    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc" event={"ID":"ebfc3fe3-2201-4b00-9342-621f4d96e281","Type":"ContainerDied","Data":"1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51"}
Oct 30 08:02:21.745789 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.745683    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fdccdf0c8e5c369033ed9495e41d75f352a23a6a984f2ac9ef0eae771664a51"
Oct 30 08:02:21.745789 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:21.745769    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"
Oct 30 08:02:22.966211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:22.966180    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 08:02:22.992354 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:22.992322    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 08:02:23.064948 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.064912    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\") pod \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") "
Oct 30 08:02:23.065170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.065050    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\" (UniqueName: \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\") pod \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\" (UID: \"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4\") "
Oct 30 08:02:23.065398 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.065368    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume" pod "6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" (UID: "6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:23.066039 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.066015    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:23.068810 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.068775    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb" pod "6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" (UID: "6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4"). InnerVolumeSpecName "kube-api-access-qlvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:23.166662 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.166631    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\") pod \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") "
Oct 30 08:02:23.166896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.166883    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\" (UniqueName: \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\") pod \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\" (UID: \"ef2e9c97-5e4f-4081-ab46-537d26cf5554\") "
Oct 30 08:02:23.167030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.167008    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume" pod "ef2e9c97-5e4f-4081-ab46-537d26cf5554" (UID: "ef2e9c97-5e4f-4081-ab46-537d26cf5554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:23.167322 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.167304    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2e9c97-5e4f-4081-ab46-537d26cf5554-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:23.167322 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.167322    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlvjb\" (UniqueName: \"kubernetes.io/projected/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4-kube-api-access-qlvjb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:23.169026 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.168997    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5" pod "ef2e9c97-5e4f-4081-ab46-537d26cf5554" (UID: "ef2e9c97-5e4f-4081-ab46-537d26cf5554"). InnerVolumeSpecName "kube-api-access-q25w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:23.269025 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.268945    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q25w5\" (UniqueName: \"kubernetes.io/projected/ef2e9c97-5e4f-4081-ab46-537d26cf5554-kube-api-access-q25w5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:23.404702 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.404658    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 08:02:23.405035 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.405008    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" podUID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" containerName="maas-api" containerID="cri-o://17035223141b959069c25c4c0094e1fdc54f83dd1006e2a20ea36a89f1c2f9d5" gracePeriod=30
Oct 30 08:02:23.470122 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.470081    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:23.765962 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.765928    6738 generic.go:356] "Generic (PLEG): container finished" podID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerID="0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df" exitCode=0
Oct 30 08:02:23.766145 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.765973    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"
Oct 30 08:02:23.766145 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.766018    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" event={"ID":"ef2e9c97-5e4f-4081-ab46-537d26cf5554","Type":"ContainerDied","Data":"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"}
Oct 30 08:02:23.766145 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.766079    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z" event={"ID":"ef2e9c97-5e4f-4081-ab46-537d26cf5554","Type":"ContainerDied","Data":"18a3ebbaf2f018e785344579511f0d1f95ef6e38c043c54eb2633de42f632582"}
Oct 30 08:02:23.766145 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.766108    6738 scope.go:122] "RemoveContainer" containerID="0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"
Oct 30 08:02:23.767756 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.767726    6738 generic.go:356] "Generic (PLEG): container finished" podID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" containerID="17035223141b959069c25c4c0094e1fdc54f83dd1006e2a20ea36a89f1c2f9d5" exitCode=0
Oct 30 08:02:23.767863 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.767807    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" event={"ID":"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11","Type":"ContainerDied","Data":"17035223141b959069c25c4c0094e1fdc54f83dd1006e2a20ea36a89f1c2f9d5"}
Oct 30 08:02:23.768243 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.767878    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94" event={"ID":"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11","Type":"ContainerDied","Data":"fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0"}
Oct 30 08:02:23.768376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.768274    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9349afa7d450f34f40d0ec96ea6ed34726c85f5ab4e14245cf15dc339a01d0"
Oct 30 08:02:23.769586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.769563    6738 generic.go:356] "Generic (PLEG): container finished" podID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerID="b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0" exitCode=0
Oct 30 08:02:23.769586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.769604    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" event={"ID":"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4","Type":"ContainerDied","Data":"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"}
Oct 30 08:02:23.769809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.769627    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"
Oct 30 08:02:23.769809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.769642    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn" event={"ID":"6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4","Type":"ContainerDied","Data":"dd8e053f3dfb4cbbdcc31078ea3c08ee5ea36266d2f579c59e8a8b0ff2b42f73"}
Oct 30 08:02:23.784774 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.784753    6738 scope.go:122] "RemoveContainer" containerID="0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"
Oct 30 08:02:23.785209 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:23.785179    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df\": container with ID starting with 0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df not found: ID does not exist" containerID="0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"
Oct 30 08:02:23.785335 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.785218    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df"} err="failed to get container status \"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df\": rpc error: code = NotFound desc = could not find container \"0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df\": container with ID starting with 0e5776f39040bdae4c29b3c6a4a104c32fecd2ccf6477da559e1e6f2a836c0df not found: ID does not exist"
Oct 30 08:02:23.785335 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.785235    6738 scope.go:122] "RemoveContainer" containerID="b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"
Oct 30 08:02:23.785953 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.785928    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 08:02:23.799015 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.798984    6738 scope.go:122] "RemoveContainer" containerID="b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"
Oct 30 08:02:23.799523 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:23.799498    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0\": container with ID starting with b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0 not found: ID does not exist" containerID="b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"
Oct 30 08:02:23.799630 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.799533    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0"} err="failed to get container status \"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0\": rpc error: code = NotFound desc = could not find container \"b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0\": container with ID starting with b48b1d4a352a3e297f6e0bfc340fce0cc6f158316353655439fc53a16722d5f0 not found: ID does not exist"
Oct 30 08:02:23.915661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.912545    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\") pod \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") "
Oct 30 08:02:23.915661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.912694    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\") pod \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") "
Oct 30 08:02:23.915661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.912775    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\" (UniqueName: \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\") pod \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") "
Oct 30 08:02:23.915661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.912884    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\") pod \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\" (UID: \"75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11\") "
Oct 30 08:02:23.927336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:23.925501    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca" pod "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" (UID: "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:24.016554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.011717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls" pod "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" (UID: "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:24.016554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.011775    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 08:02:24.017080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.016847    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:24.017080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.016877    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:24.017080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.016906    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bec5c-d99879487-lq9dn"]
Oct 30 08:02:24.022541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.022051    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2" pod "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" (UID: "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11"). InnerVolumeSpecName "kube-api-access-4c6k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:24.022541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.022442    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs" pod "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" (UID: "75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:24.022541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.022490    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 08:02:24.025357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.024154    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bec5c-74989c7bf8-7zn4z"]
Oct 30 08:02:24.118358 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.118239    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:24.118358 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.118269    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6k2\" (UniqueName: \"kubernetes.io/projected/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11-kube-api-access-4c6k2\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:24.206435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.206386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:24.316468 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.316433    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"]
Oct 30 08:02:24.319509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.319482    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d09eb6-gvnbl"]
Oct 30 08:02:24.324049 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.324011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:24.781569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.781541    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"
Oct 30 08:02:24.819587 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.819555    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:24.845592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.845555    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 08:02:24.855432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:24.855402    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d09eb6-b55f67567-s4b94"]
Oct 30 08:02:25.691973 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:25.691927    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204dad49-5512-4e70-99f2-b6bb0657eb5e" path="/var/lib/kubelet/pods/204dad49-5512-4e70-99f2-b6bb0657eb5e/volumes"
Oct 30 08:02:25.692506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:25.692435    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" path="/var/lib/kubelet/pods/6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4/volumes"
Oct 30 08:02:25.692902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:25.692885    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" path="/var/lib/kubelet/pods/75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11/volumes"
Oct 30 08:02:25.693402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:25.693377    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" path="/var/lib/kubelet/pods/ef2e9c97-5e4f-4081-ab46-537d26cf5554/volumes"
Oct 30 08:02:26.585903 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:26.585867    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:28.479092 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:28.479027    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:28.616983 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:28.616944    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:28.744851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:28.744755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:30.126811 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.126763    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:30.722230 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.722191    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:30.840164 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.840130    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:30.910046 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.910007    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"]
Oct 30 08:02:30.922210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.922171    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-265384-4lv55"]
Oct 30 08:02:30.987464 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:30.987360    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:31.248248 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.248161    6738 scope.go:122] "RemoveContainer" containerID="4576154abe4590b2920fd1967d2e7f7a7e8ff97234139894549415df435ec712"
Oct 30 08:02:31.259601 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.259568    6738 scope.go:122] "RemoveContainer" containerID="ab3c0cc1425afdbe86b85779f4fbd981bd54bab75a1008ddc85d4386ae45acaa"
Oct 30 08:02:31.270894 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.270865    6738 scope.go:122] "RemoveContainer" containerID="17035223141b959069c25c4c0094e1fdc54f83dd1006e2a20ea36a89f1c2f9d5"
Oct 30 08:02:31.280961 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.280940    6738 scope.go:122] "RemoveContainer" containerID="fb91bb3d94bff17061a4a7d3c52f91866e79644599fab0110dd89c1c22648e56"
Oct 30 08:02:31.290775 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.290752    6738 scope.go:122] "RemoveContainer" containerID="6d637d36ff6dcd40536f634022213c2674aeb59ac5cdc2ebf18f2c9a1ea13fa3"
Oct 30 08:02:31.428753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.428711    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:31.691121 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.691086    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f232b311-a183-42c3-bbbe-59e4a95525c3" path="/var/lib/kubelet/pods/f232b311-a183-42c3-bbbe-59e4a95525c3/volumes"
Oct 30 08:02:31.792487 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.792453    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:31.914937 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:31.914895    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:32.174509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:32.174478    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:32.721175 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:32.721143    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:33.070197 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070105    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"]
Oct 30 08:02:33.070838 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070816    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070839    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070885    6738 cpu_manager.go:414] "containerMap: removing container" podUID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" containerName="maas-api"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070894    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" containerName="maas-api"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070917    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070925    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070948    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ebfc3fe3-2201-4b00-9342-621f4d96e281" containerName="revoke-keys"
Oct 30 08:02:33.070969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.070955    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfc3fe3-2201-4b00-9342-621f4d96e281" containerName="revoke-keys"
Oct 30 08:02:33.071376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.071115    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" containerName="payload-processing"
Oct 30 08:02:33.071376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.071136    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f92c2c-e3f2-4cd4-a4cf-d08cfeb6ae11" containerName="maas-api"
Oct 30 08:02:33.071376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.071149    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9e53ef-ecb3-4f28-ba98-8c9b61bd64b4" containerName="payload-pre-processing"
Oct 30 08:02:33.071376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.071162    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfc3fe3-2201-4b00-9342-621f4d96e281" containerName="revoke-keys"
Oct 30 08:02:33.077168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.077141    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.110999 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.110970    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"]
Oct 30 08:02:33.119518 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.119488    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:02:33.119694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.119562    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:02:33.219112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.219075    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwjh\" (UniqueName: \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.219277 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.219219    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.219427 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.219410    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.331820 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.331719    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwjh\" (UniqueName: \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.351493 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.351453    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.351966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.351942    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.353619 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.353441    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.388348 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.388314    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwjh\" (UniqueName: \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.407316 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.406466    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:33.437217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.437180    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:33.706318 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:33.706218    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:34.189913 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:34.189878    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919d1887_6598_41ed_a073_2821d0f2c38a.slice/crio-d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0 WatchSource:0}: Error finding container d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0: Status 404 returned error can't find the container with id d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0
Oct 30 08:02:34.190739 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:34.190718    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"]
Oct 30 08:02:34.901669 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:34.901633    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd" event={"ID":"919d1887-6598-41ed-a073-2821d0f2c38a","Type":"ContainerStarted","Data":"3ebf52853ef2c165dc8e857ba5a12cdf6e3f85bd548eb9bead6a89cdf55bf854"}
Oct 30 08:02:34.901897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:34.901690    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd" event={"ID":"919d1887-6598-41ed-a073-2821d0f2c38a","Type":"ContainerStarted","Data":"d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0"}
Oct 30 08:02:34.945680 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:34.945615    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd" podStartSLOduration=1.9456006559999999 podStartE2EDuration="1.945600656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:34.942023074 +0000 UTC m=+3915.374786967" watchObservedRunningTime="2026-10-30 08:02:34.945600656 +0000 UTC m=+3915.378364553"
Oct 30 08:02:35.781446 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:35.781405    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:35.898329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:35.898284    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:35.917062 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:35.917023    6738 generic.go:356] "Generic (PLEG): container finished" podID="919d1887-6598-41ed-a073-2821d0f2c38a" containerID="3ebf52853ef2c165dc8e857ba5a12cdf6e3f85bd548eb9bead6a89cdf55bf854" exitCode=0
Oct 30 08:02:35.917226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:35.917109    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd" event={"ID":"919d1887-6598-41ed-a073-2821d0f2c38a","Type":"ContainerDied","Data":"3ebf52853ef2c165dc8e857ba5a12cdf6e3f85bd548eb9bead6a89cdf55bf854"}
Oct 30 08:02:37.086670 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.086643    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:37.142910 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.142881    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\" (UniqueName: \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\") pod \"919d1887-6598-41ed-a073-2821d0f2c38a\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") "
Oct 30 08:02:37.143112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.142927    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\") pod \"919d1887-6598-41ed-a073-2821d0f2c38a\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") "
Oct 30 08:02:37.143112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.143053    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\") pod \"919d1887-6598-41ed-a073-2821d0f2c38a\" (UID: \"919d1887-6598-41ed-a073-2821d0f2c38a\") "
Oct 30 08:02:37.143662 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.143622    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca" pod "919d1887-6598-41ed-a073-2821d0f2c38a" (UID: "919d1887-6598-41ed-a073-2821d0f2c38a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:37.143857 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.143796    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:37.145332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.145277    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca" pod "919d1887-6598-41ed-a073-2821d0f2c38a" (UID: "919d1887-6598-41ed-a073-2821d0f2c38a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:37.148738 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.148704    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh" pod "919d1887-6598-41ed-a073-2821d0f2c38a" (UID: "919d1887-6598-41ed-a073-2821d0f2c38a"). InnerVolumeSpecName "kube-api-access-xmwjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:37.245595 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.245511    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919d1887-6598-41ed-a073-2821d0f2c38a-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:37.245595 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.245541    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwjh\" (UniqueName: \"kubernetes.io/projected/919d1887-6598-41ed-a073-2821d0f2c38a-kube-api-access-xmwjh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:37.520235 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:37.520144    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:02:37.751540 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.751506    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:37.893475 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.893432    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:37.933731 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.933702    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"
Oct 30 08:02:37.933947 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.933744    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd" event={"ID":"919d1887-6598-41ed-a073-2821d0f2c38a","Type":"ContainerDied","Data":"d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0"}
Oct 30 08:02:37.933947 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:37.933799    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60382420540dbfc14a1c3b09b856e1a8842c9dccb86f3aec5a6d3d3ee84fec0"
Oct 30 08:02:39.120073 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:39.120041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:39.236593 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:39.236557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:39.485968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:39.485930    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:39.656354 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:39.650852    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:40.085776 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.085739    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"]
Oct 30 08:02:40.086547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.086526    6738 cpu_manager.go:414] "containerMap: removing container" podUID="919d1887-6598-41ed-a073-2821d0f2c38a" containerName="revoke-keys"
Oct 30 08:02:40.086603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.086548    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919d1887-6598-41ed-a073-2821d0f2c38a" containerName="revoke-keys"
Oct 30 08:02:40.086665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.086656    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="919d1887-6598-41ed-a073-2821d0f2c38a" containerName="revoke-keys"
Oct 30 08:02:40.090387 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.090364    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.106846 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.106817    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"]
Oct 30 08:02:40.174102 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.174068    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:02:40.174633 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.174127    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:02:40.190402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.190365    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.190568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.190536    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.190619 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.190592    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72cv\" (UniqueName: \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.291967 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.291928    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.292158 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.291977    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72cv\" (UniqueName: \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.292158 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.292089    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.292822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.292798    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.294458 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.294438    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.316006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.315978    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72cv\" (UniqueName: \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.408520 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.408483    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:40.715013 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.714983    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"]
Oct 30 08:02:40.715367 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:40.715340    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa941b88_17d3_46ef_ba5e_de7abc3d4e50.slice/crio-6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c WatchSource:0}: Error finding container 6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c: Status 404 returned error can't find the container with id 6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c
Oct 30 08:02:40.957143 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:40.957105    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9" event={"ID":"aa941b88-17d3-46ef-ba5e-de7abc3d4e50","Type":"ContainerStarted","Data":"6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c"}
Oct 30 08:02:41.728187 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:41.728147    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:41.965449 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:41.965413    6738 generic.go:356] "Generic (PLEG): container finished" podID="aa941b88-17d3-46ef-ba5e-de7abc3d4e50" containerID="aed6f4e9af2b09c25c202f11cdf180bf51fab40e2e642e7eb6f7216ff6662746" exitCode=0
Oct 30 08:02:41.965638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:41.965501    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9" event={"ID":"aa941b88-17d3-46ef-ba5e-de7abc3d4e50","Type":"ContainerDied","Data":"aed6f4e9af2b09c25c202f11cdf180bf51fab40e2e642e7eb6f7216ff6662746"}
Oct 30 08:02:43.117690 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.117660    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:43.232859 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.232773    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\") pod \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") "
Oct 30 08:02:43.232859 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.232852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\") pod \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") "
Oct 30 08:02:43.233133 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.232951    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\" (UniqueName: \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\") pod \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\" (UID: \"aa941b88-17d3-46ef-ba5e-de7abc3d4e50\") "
Oct 30 08:02:43.233356 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.233280    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca" pod "aa941b88-17d3-46ef-ba5e-de7abc3d4e50" (UID: "aa941b88-17d3-46ef-ba5e-de7abc3d4e50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:43.233554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.233533    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:43.235081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.235058    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca" pod "aa941b88-17d3-46ef-ba5e-de7abc3d4e50" (UID: "aa941b88-17d3-46ef-ba5e-de7abc3d4e50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:43.235187 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.235167    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv" pod "aa941b88-17d3-46ef-ba5e-de7abc3d4e50" (UID: "aa941b88-17d3-46ef-ba5e-de7abc3d4e50"). InnerVolumeSpecName "kube-api-access-p72cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:43.334400 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.334354    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:43.334400 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.334390    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72cv\" (UniqueName: \"kubernetes.io/projected/aa941b88-17d3-46ef-ba5e-de7abc3d4e50-kube-api-access-p72cv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:43.476070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.476029    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"]
Oct 30 08:02:43.476913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.476877    6738 cpu_manager.go:414] "containerMap: removing container" podUID="aa941b88-17d3-46ef-ba5e-de7abc3d4e50" containerName="revoke-keys"
Oct 30 08:02:43.476913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.476903    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa941b88-17d3-46ef-ba5e-de7abc3d4e50" containerName="revoke-keys"
Oct 30 08:02:43.477151 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.477105    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa941b88-17d3-46ef-ba5e-de7abc3d4e50" containerName="revoke-keys"
Oct 30 08:02:43.480861 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.480837    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.504149 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.504111    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"]
Oct 30 08:02:43.537813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.537723    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.537813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.537811    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9l27\" (UniqueName: \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.538094 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.537955    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.570526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.570484    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"]
Oct 30 08:02:43.574871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.574848    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.599683 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.599649    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"]
Oct 30 08:02:43.636535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.636500    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 08:02:43.639578 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.639550    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmd8\" (UniqueName: \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.639733 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.639670    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.639806 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.639745    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l27\" (UniqueName: \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.639900 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.639887    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.640110 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.639954    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.640110 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.640054    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.640640 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.640619    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.642452 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.642426    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.663931 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.663894    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9l27\" (UniqueName: \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\") pod \"maas-api-revoke-sub-4818406e-gjms9\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.741705 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.741604    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.741705 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.741684    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.741985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.741744    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmd8\" (UniqueName: \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.743021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.742988    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.745951 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.745918    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.767235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.767202    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmd8\" (UniqueName: \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\") pod \"maas-api-revoke-sub-c1600e8f-pg9gd\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.775423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.775389    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:43.795714 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.795681    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:43.899784 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.899727    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:43.937526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.937492    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:44.004666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.997849    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9" event={"ID":"aa941b88-17d3-46ef-ba5e-de7abc3d4e50","Type":"ContainerDied","Data":"6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c"}
Oct 30 08:02:44.004666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.998284    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6705b6a93e479b818a8499ef0e0bad92e38200266bf80636cb66136cade5029c"
Oct 30 08:02:44.004666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:43.998351    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"
Oct 30 08:02:44.073659 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:44.073623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:44.142226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:44.142197    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"]
Oct 30 08:02:44.142601 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:44.142309    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0740f920_de80_4520_8801_3a4babbcc199.slice/crio-b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb WatchSource:0}: Error finding container b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb: Status 404 returned error can't find the container with id b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb
Oct 30 08:02:44.295379 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:44.292220    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"]
Oct 30 08:02:44.364856 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:44.364738    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8d20d6_58fd_4888_b638_135de8805efb.slice/crio-07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c WatchSource:0}: Error finding container 07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c: Status 404 returned error can't find the container with id 07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c
Oct 30 08:02:45.006721 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.006672    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9" event={"ID":"0740f920-de80-4520-8801-3a4babbcc199","Type":"ContainerStarted","Data":"1eacf816f57be7ef76b53b1af703227159d9f3f55816ebf0b46d5c0cad7ff793"}
Oct 30 08:02:45.006917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.006779    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9" event={"ID":"0740f920-de80-4520-8801-3a4babbcc199","Type":"ContainerStarted","Data":"b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb"}
Oct 30 08:02:45.008540 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.008497    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd" event={"ID":"5f8d20d6-58fd-4888-b638-135de8805efb","Type":"ContainerStarted","Data":"3e7c7fffa40979f40109d48ca357fb95177ea93a2b0c887bfcc4fb817d661784"}
Oct 30 08:02:45.008682 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.008550    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd" event={"ID":"5f8d20d6-58fd-4888-b638-135de8805efb","Type":"ContainerStarted","Data":"07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c"}
Oct 30 08:02:45.049885 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.049810    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9" podStartSLOduration=2.049796455 podStartE2EDuration="2.049796455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:45.046897646 +0000 UTC m=+3925.479661539" watchObservedRunningTime="2026-10-30 08:02:45.049796455 +0000 UTC m=+3925.482560348"
Oct 30 08:02:45.091218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.091156    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd" podStartSLOduration=2.09113617 podStartE2EDuration="2.09113617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:45.08913711 +0000 UTC m=+3925.521901002" watchObservedRunningTime="2026-10-30 08:02:45.09113617 +0000 UTC m=+3925.523900065"
Oct 30 08:02:45.907182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:45.907148    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:46.018154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:46.018118    6738 generic.go:356] "Generic (PLEG): container finished" podID="0740f920-de80-4520-8801-3a4babbcc199" containerID="1eacf816f57be7ef76b53b1af703227159d9f3f55816ebf0b46d5c0cad7ff793" exitCode=0
Oct 30 08:02:46.018382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:46.018209    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9" event={"ID":"0740f920-de80-4520-8801-3a4babbcc199","Type":"ContainerDied","Data":"1eacf816f57be7ef76b53b1af703227159d9f3f55816ebf0b46d5c0cad7ff793"}
Oct 30 08:02:46.019989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:46.019964    6738 generic.go:356] "Generic (PLEG): container finished" podID="5f8d20d6-58fd-4888-b638-135de8805efb" containerID="3e7c7fffa40979f40109d48ca357fb95177ea93a2b0c887bfcc4fb817d661784" exitCode=0
Oct 30 08:02:46.020114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:46.020043    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd" event={"ID":"5f8d20d6-58fd-4888-b638-135de8805efb","Type":"ContainerDied","Data":"3e7c7fffa40979f40109d48ca357fb95177ea93a2b0c887bfcc4fb817d661784"}
Oct 30 08:02:47.193162 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.193126    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:47.257034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.257005    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:47.330565 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330466    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\") pod \"5f8d20d6-58fd-4888-b638-135de8805efb\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") "
Oct 30 08:02:47.330778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330590    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\") pod \"0740f920-de80-4520-8801-3a4babbcc199\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") "
Oct 30 08:02:47.330778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330639    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\" (UniqueName: \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\") pod \"5f8d20d6-58fd-4888-b638-135de8805efb\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") "
Oct 30 08:02:47.330778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330671    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\") pod \"5f8d20d6-58fd-4888-b638-135de8805efb\" (UID: \"5f8d20d6-58fd-4888-b638-135de8805efb\") "
Oct 30 08:02:47.330984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330796    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\") pod \"0740f920-de80-4520-8801-3a4babbcc199\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") "
Oct 30 08:02:47.330984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330908    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\" (UniqueName: \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\") pod \"0740f920-de80-4520-8801-3a4babbcc199\" (UID: \"0740f920-de80-4520-8801-3a4babbcc199\") "
Oct 30 08:02:47.330984 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.330939    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca" pod "5f8d20d6-58fd-4888-b638-135de8805efb" (UID: "5f8d20d6-58fd-4888-b638-135de8805efb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:47.331594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.331359    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca" pod "0740f920-de80-4520-8801-3a4babbcc199" (UID: "0740f920-de80-4520-8801-3a4babbcc199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:47.331594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.331549    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.331594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.331568    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0740f920-de80-4520-8801-3a4babbcc199-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.333420 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.333320    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca" pod "0740f920-de80-4520-8801-3a4babbcc199" (UID: "0740f920-de80-4520-8801-3a4babbcc199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:47.333750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.333720    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca" pod "5f8d20d6-58fd-4888-b638-135de8805efb" (UID: "5f8d20d6-58fd-4888-b638-135de8805efb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:47.334716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.334689    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27" pod "0740f920-de80-4520-8801-3a4babbcc199" (UID: "0740f920-de80-4520-8801-3a4babbcc199"). InnerVolumeSpecName "kube-api-access-x9l27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:47.334853 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.334737    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8" pod "5f8d20d6-58fd-4888-b638-135de8805efb" (UID: "5f8d20d6-58fd-4888-b638-135de8805efb"). InnerVolumeSpecName "kube-api-access-8mmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:47.433289 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.433253    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0740f920-de80-4520-8801-3a4babbcc199-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.433511 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.433289    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmd8\" (UniqueName: \"kubernetes.io/projected/5f8d20d6-58fd-4888-b638-135de8805efb-kube-api-access-8mmd8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.433511 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.433346    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f8d20d6-58fd-4888-b638-135de8805efb-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.433511 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.433361    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9l27\" (UniqueName: \"kubernetes.io/projected/0740f920-de80-4520-8801-3a4babbcc199-kube-api-access-x9l27\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:47.753830 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:47.753792    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:48.037192 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.037102    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"
Oct 30 08:02:48.037192 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.037133    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd" event={"ID":"5f8d20d6-58fd-4888-b638-135de8805efb","Type":"ContainerDied","Data":"07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c"}
Oct 30 08:02:48.037192 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.037186    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07434b3030939846afda4b7cfa4a7295af35b7eeed9a8bb3fece929f66af441c"
Oct 30 08:02:48.038895 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.038870    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"
Oct 30 08:02:48.039055 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.038898    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9" event={"ID":"0740f920-de80-4520-8801-3a4babbcc199","Type":"ContainerDied","Data":"b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb"}
Oct 30 08:02:48.039055 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.038943    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b478775c5b421aa0e5068ace2ca91cbc6dc4471d1c9e5e730897f14a444061cb"
Oct 30 08:02:48.637317 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.637228    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" podUID="e96d55e9-2df2-42ab-b495-e89dfeff776a" containerName="payload-pre-processing" containerID="cri-o://3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b" gracePeriod=25
Oct 30 08:02:48.913398 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:48.913287    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 08:02:49.055998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.055960    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\") pod \"e96d55e9-2df2-42ab-b495-e89dfeff776a\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") "
Oct 30 08:02:49.056212 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.056009    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\" (UniqueName: \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\") pod \"e96d55e9-2df2-42ab-b495-e89dfeff776a\" (UID: \"e96d55e9-2df2-42ab-b495-e89dfeff776a\") "
Oct 30 08:02:49.056406 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.056379    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume" pod "e96d55e9-2df2-42ab-b495-e89dfeff776a" (UID: "e96d55e9-2df2-42ab-b495-e89dfeff776a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:49.056504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.056490    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96d55e9-2df2-42ab-b495-e89dfeff776a-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.058338 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.058282    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m" pod "e96d55e9-2df2-42ab-b495-e89dfeff776a" (UID: "e96d55e9-2df2-42ab-b495-e89dfeff776a"). InnerVolumeSpecName "kube-api-access-x6k2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:49.059077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.059050    6738 generic.go:356] "Generic (PLEG): container finished" podID="e96d55e9-2df2-42ab-b495-e89dfeff776a" containerID="3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b" exitCode=0
Oct 30 08:02:49.059168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.059116    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"
Oct 30 08:02:49.059251 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.059118    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" event={"ID":"e96d55e9-2df2-42ab-b495-e89dfeff776a","Type":"ContainerDied","Data":"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"}
Oct 30 08:02:49.059347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.059260    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm" event={"ID":"e96d55e9-2df2-42ab-b495-e89dfeff776a","Type":"ContainerDied","Data":"1265f19b9850ba138388f628a8a111d39a1a48399049d4c4e110b174e8ac977b"}
Oct 30 08:02:49.059347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.059281    6738 scope.go:122] "RemoveContainer" containerID="3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"
Oct 30 08:02:49.076210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.076191    6738 scope.go:122] "RemoveContainer" containerID="3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"
Oct 30 08:02:49.076572 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:02:49.076552    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b\": container with ID starting with 3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b not found: ID does not exist" containerID="3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"
Oct 30 08:02:49.076634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.076581    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b"} err="failed to get container status \"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b\": rpc error: code = NotFound desc = could not find container \"3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b\": container with ID starting with 3343ca88e1fc37461fef75f81a6767826540e5a783e06c4d21ded9889e87d01b not found: ID does not exist"
Oct 30 08:02:49.115099 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.115060    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 08:02:49.123896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.123866    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bab4c-5675bdc455-tjbcm"]
Oct 30 08:02:49.158135 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.158052    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6k2m\" (UniqueName: \"kubernetes.io/projected/e96d55e9-2df2-42ab-b495-e89dfeff776a-kube-api-access-x6k2m\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.369718 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.369684    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:49.694572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.694535    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96d55e9-2df2-42ab-b495-e89dfeff776a" path="/var/lib/kubelet/pods/e96d55e9-2df2-42ab-b495-e89dfeff776a/volumes"
Oct 30 08:02:49.990703 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:49.990597    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:50.108738 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.108698    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:50.141569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.141530    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"]
Oct 30 08:02:50.153134 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.153101    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c1b5d7-cvhkc"]
Oct 30 08:02:50.238334 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.229165    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:50.646961 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.646916    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 08:02:50.660587 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.660554    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:50.840821 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.840788    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:50.964830 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:50.964727    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:51.690307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:51.690268    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfc3fe3-2201-4b00-9342-621f4d96e281" path="/var/lib/kubelet/pods/ebfc3fe3-2201-4b00-9342-621f4d96e281/volumes"
Oct 30 08:02:52.249394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:52.249354    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:53.777040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:53.776986    6738 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podef2e9c97-5e4f-4081-ab46-537d26cf5554"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podef2e9c97-5e4f-4081-ab46-537d26cf5554] : Timed out while waiting for systemd to remove kubepods-burstable-podef2e9c97_5e4f_4081_ab46_537d26cf5554.slice"
Oct 30 08:02:53.777491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:53.777040    6738 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="ef2e9c97-5e4f-4081-ab46-537d26cf5554" err="failed to delete cgroup paths for [kubepods burstable podef2e9c97-5e4f-4081-ab46-537d26cf5554] : unable to destroy cgroup paths for cgroup [kubepods burstable podef2e9c97-5e4f-4081-ab46-537d26cf5554] : Timed out while waiting for systemd to remove kubepods-burstable-podef2e9c97_5e4f_4081_ab46_537d26cf5554.slice"
Oct 30 08:02:54.203779 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:54.203743    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:55.654555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:55.654424    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing" containerID="cri-o://73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c" gracePeriod=25
Oct 30 08:02:56.145681 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:56.145646    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:57.609628 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:57.609590    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:57.735860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:57.735816    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:57.933134 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:57.933099    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 08:02:57.933399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:57.933329    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy" containerID="cri-o://a87e8fad2404ab059324a0a1cb7dc3b159b27d7a34e20768696829a3e3b13d58" gracePeriod=30
Oct 30 08:02:57.967407 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:57.967370    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:58.085392 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.085361    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:02:58.518957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.518923    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"]
Oct 30 08:02:58.519585 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519570    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5f8d20d6-58fd-4888-b638-135de8805efb" containerName="revoke-keys"
Oct 30 08:02:58.519651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519587    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8d20d6-58fd-4888-b638-135de8805efb" containerName="revoke-keys"
Oct 30 08:02:58.519651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519611    6738 cpu_manager.go:414] "containerMap: removing container" podUID="0740f920-de80-4520-8801-3a4babbcc199" containerName="revoke-keys"
Oct 30 08:02:58.519651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519616    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0740f920-de80-4520-8801-3a4babbcc199" containerName="revoke-keys"
Oct 30 08:02:58.519651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519646    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e96d55e9-2df2-42ab-b495-e89dfeff776a" containerName="payload-pre-processing"
Oct 30 08:02:58.519651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519651    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96d55e9-2df2-42ab-b495-e89dfeff776a" containerName="payload-pre-processing"
Oct 30 08:02:58.519822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519731    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8d20d6-58fd-4888-b638-135de8805efb" containerName="revoke-keys"
Oct 30 08:02:58.519822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519744    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="0740f920-de80-4520-8801-3a4babbcc199" containerName="revoke-keys"
Oct 30 08:02:58.519822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.519755    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96d55e9-2df2-42ab-b495-e89dfeff776a" containerName="payload-pre-processing"
Oct 30 08:02:58.523368 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.523351    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.544060 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.544031    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"]
Oct 30 08:02:58.635617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.635580    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:02:58.635617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.635616    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:02:58.653399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.653362    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.653549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.653496    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.653596 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.653581    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvcj\" (UniqueName: \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.755716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.755670    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.755926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.755847    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.755995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.755971    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvcj\" (UniqueName: \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.767451 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.759463    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.768771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.768739    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.794736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.794657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvcj\" (UniqueName: \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:58.837975 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:58.837940    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:02:59.148006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:59.147811    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:59.148148 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:02:59.148055    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b107f03_17cb_4fc1_ae26_5ae5cee9c1f1.slice/crio-ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466 WatchSource:0}: Error finding container ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466: Status 404 returned error can't find the container with id ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466
Oct 30 08:02:59.148673 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:59.148607    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"]
Oct 30 08:02:59.164526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:02:59.164489    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6" event={"ID":"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1","Type":"ContainerStarted","Data":"ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466"}
Oct 30 08:03:00.173265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:00.173230    6738 generic.go:356] "Generic (PLEG): container finished" podID="6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" containerID="8cc7eb1e598725acd1576d40e9750c81eb0278cf7ba7feb4910ec084d88c55cf" exitCode=0
Oct 30 08:03:00.173759 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:00.173323    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6" event={"ID":"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1","Type":"ContainerDied","Data":"8cc7eb1e598725acd1576d40e9750c81eb0278cf7ba7feb4910ec084d88c55cf"}
Oct 30 08:03:00.193496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:00.193461    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:01.352200 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.352171    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:03:01.491704 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.491617    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\") pod \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") "
Oct 30 08:03:01.491910 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.491728    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\" (UniqueName: \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\") pod \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") "
Oct 30 08:03:01.491910 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.491828    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\") pod \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\" (UID: \"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1\") "
Oct 30 08:03:01.492229 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.492203    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca" pod "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" (UID: "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:01.493748 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.493722    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca" pod "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" (UID: "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:01.493888 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.493864    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj" pod "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" (UID: "6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1"). InnerVolumeSpecName "kube-api-access-vgvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:01.593472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.593428    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:01.593472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.593466    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgvcj\" (UniqueName: \"kubernetes.io/projected/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-kube-api-access-vgvcj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:01.593472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:01.593477    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:02.124321 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:02.124239    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:02.208734 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:02.208706    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"
Oct 30 08:03:02.208959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:02.208708    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6" event={"ID":"6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1","Type":"ContainerDied","Data":"ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466"}
Oct 30 08:03:02.208959 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:02.208833    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba720f5e73575d76ae66196de629af03d7f55399cede09613dc2079505294466"
Oct 30 08:03:03.220353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.220284    6738 generic.go:356] "Generic (PLEG): container finished" podID="441fa007-4793-4b3c-b75b-51316af6d81e" containerID="a87e8fad2404ab059324a0a1cb7dc3b159b27d7a34e20768696829a3e3b13d58" exitCode=0
Oct 30 08:03:03.220722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.220321    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerDied","Data":"a87e8fad2404ab059324a0a1cb7dc3b159b27d7a34e20768696829a3e3b13d58"}
Oct 30 08:03:03.220722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.220476    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q" event={"ID":"441fa007-4793-4b3c-b75b-51316af6d81e","Type":"ContainerDied","Data":"1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b"}
Oct 30 08:03:03.220722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.220505    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a85a5b3064825f7018cf309622e7cb0ade3a450bc2dfb1678144e7d8d2c009b"
Oct 30 08:03:03.220722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.220519    6738 scope.go:122] "RemoveContainer" containerID="057e748f8e403a9bf64537a809e0e95a0e590af58fd43af815c59b4f7f8c8a82"
Oct 30 08:03:03.225737 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.225711    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 08:03:03.321834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.321791    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.321848    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.321892    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.321943    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322070 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322057    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322146    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:03.322497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322343    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:03.322497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322392    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:03.322497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322405    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:03.322497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322264    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322521    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322556    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.322728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.322597    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\") pod \"441fa007-4793-4b3c-b75b-51316af6d81e\" (UID: \"441fa007-4793-4b3c-b75b-51316af6d81e\") "
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323391    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323660    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441fa007-4793-4b3c-b75b-51316af6d81e-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323681    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323695    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323709    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.323760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.323724    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.325255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.324922    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:03:03.325255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.325073    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "kube-api-access-sj424". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:03.325255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.325149    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:03.325687 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.325664    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token" pod "441fa007-4793-4b3c-b75b-51316af6d81e" (UID: "441fa007-4793-4b3c-b75b-51316af6d81e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:03.425264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.425231    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441fa007-4793-4b3c-b75b-51316af6d81e-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.425264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.425263    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.425502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.425276    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj424\" (UniqueName: \"kubernetes.io/projected/441fa007-4793-4b3c-b75b-51316af6d81e-kube-api-access-sj424\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.425502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.425287    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441fa007-4793-4b3c-b75b-51316af6d81e-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:03.930864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:03.930829    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:04.229851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:04.229821    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"
Oct 30 08:03:04.328535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:04.328494    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 08:03:04.338153 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:04.338120    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c1b5d7-openshift-default-5d4ff7f649-v252q"]
Oct 30 08:03:05.224798 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:05.224764    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:05.691247 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:05.691205    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" path="/var/lib/kubelet/pods/441fa007-4793-4b3c-b75b-51316af6d81e/volumes"
Oct 30 08:03:05.783981 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:05.783942    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:07.519289 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:07.519251    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:03:07.538447 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:07.538380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:09.065790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:09.065755    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 08:03:09.147941 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:09.147895    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:09.215882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:09.215845    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:09.333247 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:09.333152    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:09.482677 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:09.482636    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:11.185045 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:11.185008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:13.018820 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:13.018772    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:14.066728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:14.066678    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing" containerID="cri-o://d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6" gracePeriod=25
Oct 30 08:03:14.725812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:14.725776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:15.270955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.270883    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:15.420817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.420774    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:15.506608 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.506568    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"]
Oct 30 08:03:15.509743 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.509712    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-932c00-rcbk6"]
Oct 30 08:03:15.546473 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.546385    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:15.697881 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.697836    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" path="/var/lib/kubelet/pods/6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1/volumes"
Oct 30 08:03:15.942861 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:15.942833    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 08:03:16.040332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.031815    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:16.040332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.039761    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\" (UniqueName: \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\") pod \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") "
Oct 30 08:03:16.040332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.039923    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\") pod \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\" (UID: \"2d4a4a3c-565f-4ffc-ad46-020c713c3f51\") "
Oct 30 08:03:16.066328 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.064548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume" pod "2d4a4a3c-565f-4ffc-ad46-020c713c3f51" (UID: "2d4a4a3c-565f-4ffc-ad46-020c713c3f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:16.074835 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.074329    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz" pod "2d4a4a3c-565f-4ffc-ad46-020c713c3f51" (UID: "2d4a4a3c-565f-4ffc-ad46-020c713c3f51"). InnerVolumeSpecName "kube-api-access-qv8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:16.141355 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.141289    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8kz\" (UniqueName: \"kubernetes.io/projected/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-kube-api-access-qv8kz\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:16.141355 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.141340    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4a4a3c-565f-4ffc-ad46-020c713c3f51-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:16.157533 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.157501    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:16.270999 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.270960    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:16.310957 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.310923    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"]
Oct 30 08:03:16.315398 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.315360    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-06e19a-r2sm9"]
Oct 30 08:03:16.327256 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.327223    6738 generic.go:356] "Generic (PLEG): container finished" podID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerID="73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c" exitCode=0
Oct 30 08:03:16.327428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.327270    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"
Oct 30 08:03:16.327428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.327339    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" event={"ID":"2d4a4a3c-565f-4ffc-ad46-020c713c3f51","Type":"ContainerDied","Data":"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"}
Oct 30 08:03:16.327428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.327391    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq" event={"ID":"2d4a4a3c-565f-4ffc-ad46-020c713c3f51","Type":"ContainerDied","Data":"abc7a97198fd42d099935020ddd0f7d56054d245763bede7de4dc6fb0977f699"}
Oct 30 08:03:16.327428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.327416    6738 scope.go:122] "RemoveContainer" containerID="73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"
Oct 30 08:03:16.338457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.338435    6738 scope.go:122] "RemoveContainer" containerID="73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"
Oct 30 08:03:16.338745 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:16.338722    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c\": container with ID starting with 73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c not found: ID does not exist" containerID="73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"
Oct 30 08:03:16.338825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.338753    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c"} err="failed to get container status \"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c\": rpc error: code = NotFound desc = could not find container \"73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c\": container with ID starting with 73ab78140922d21b05bb304b52f5fcca8b5ab607487dd4fcc4c3810ff329d91c not found: ID does not exist"
Oct 30 08:03:16.392908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.392537    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:16.407545 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.407505    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 08:03:16.419104 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.419069    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-06e19a-ff9b95984-wm2kq"]
Oct 30 08:03:16.845626 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:16.845533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:17.133059 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.133026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:17.349519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.349481    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:17.543537 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.543444    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:17.691469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.691431    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" path="/var/lib/kubelet/pods/2d4a4a3c-565f-4ffc-ad46-020c713c3f51/volumes"
Oct 30 08:03:17.692106 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.692080    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa941b88-17d3-46ef-ba5e-de7abc3d4e50" path="/var/lib/kubelet/pods/aa941b88-17d3-46ef-ba5e-de7abc3d4e50/volumes"
Oct 30 08:03:17.762654 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:17.762618    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:18.032751 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.032703    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"]
Oct 30 08:03:18.044489 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.044460    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff40750-whtpm"]
Oct 30 08:03:18.129574 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.129544    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:18.291843 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.291763    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:18.343833 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.343797    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:18.771797 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.771755    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:18.883741 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.883699    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:18.919192 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.919147    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"]
Oct 30 08:03:18.931075 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:18.931036    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2bab4c-6fwbd"]
Oct 30 08:03:19.006457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:19.006418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:19.414046 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:19.414010    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:19.694384 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:19.694284    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6585be74-ee6f-4a06-ba42-e31ec3e660ae" path="/var/lib/kubelet/pods/6585be74-ee6f-4a06-ba42-e31ec3e660ae/volumes"
Oct 30 08:03:19.695403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:19.695377    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919d1887-6598-41ed-a073-2821d0f2c38a" path="/var/lib/kubelet/pods/919d1887-6598-41ed-a073-2821d0f2c38a/volumes"
Oct 30 08:03:20.854434 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:20.854401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:22.731925 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:22.731888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:23.670526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:23.670491    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:23.796423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:23.796385    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:24.104132 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.104047    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 08:03:24.104320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.104258    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy" containerID="cri-o://5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5" gracePeriod=30
Oct 30 08:03:24.141683 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.141653    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:24.641123 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641084    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"]
Oct 30 08:03:24.641728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641714    6738 cpu_manager.go:414] "containerMap: removing container" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641729    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641743    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" containerName="revoke-keys"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641748    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" containerName="revoke-keys"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641773    6738 cpu_manager.go:414] "containerMap: removing container" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641779    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641801    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing"
Oct 30 08:03:24.641807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641806    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing"
Oct 30 08:03:24.642052 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641887    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4a4a3c-565f-4ffc-ad46-020c713c3f51" containerName="payload-processing"
Oct 30 08:03:24.642052 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641896    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:24.642052 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.641905    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b107f03-17cb-4fc1-ae26-5ae5cee9c1f1" containerName="revoke-keys"
Oct 30 08:03:24.645256 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.645233    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.664170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.664133    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"]
Oct 30 08:03:24.701490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.701456    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:03:24.701698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.701630    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:03:24.765805 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.765776    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:24.794548 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.794505    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.794790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.794679    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.794875 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.794840    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzzb\" (UniqueName: \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.890517 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.890483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:24.895925 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.895855    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.896039 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.895950    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.896088 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.896068    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzzb\" (UniqueName: \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.896576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.896555    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.898419 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.898396    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.922382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.922348    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzzb\" (UniqueName: \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:24.958746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:24.958719    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:25.089225 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.089132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:25.289478 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.289409    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:25.289825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.289690    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"]
Oct 30 08:03:25.428720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.428685    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8" event={"ID":"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d","Type":"ContainerStarted","Data":"cfb174cc506116f1b269d36a35c339f12e248fd45d5134711b43fb44c68e9b2f"}
Oct 30 08:03:25.622041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.622010    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:25.879353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.879221    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"]
Oct 30 08:03:25.880076 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.880051    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="441fa007-4793-4b3c-b75b-51316af6d81e" containerName="istio-proxy"
Oct 30 08:03:25.883857 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.883834    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:25.905604 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:25.905571    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"]
Oct 30 08:03:26.018944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.018907    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwtm\" (UniqueName: \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.019129 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.018975    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.019129 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.019112    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.120418 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.120381    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwtm\" (UniqueName: \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.120607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.120476    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.120607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.120553    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.121241 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.121221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.123101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.123078    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.144260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.144186    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwtm\" (UniqueName: \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.198213 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.198170    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:26.439118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.439024    6738 generic.go:356] "Generic (PLEG): container finished" podID="a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" containerID="65e66e34b730cdf8fc4103f64098a72a9e0949143a74ba59219977c104d463c3" exitCode=0
Oct 30 08:03:26.439118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.439104    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8" event={"ID":"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d","Type":"ContainerDied","Data":"65e66e34b730cdf8fc4103f64098a72a9e0949143a74ba59219977c104d463c3"}
Oct 30 08:03:26.505841 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:26.505809    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"]
Oct 30 08:03:26.506656 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:03:26.506614    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1304a3ec_0e83_4205_ab4b_be0d16fc7194.slice/crio-89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c WatchSource:0}: Error finding container 89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c: Status 404 returned error can't find the container with id 89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c
Oct 30 08:03:27.051014 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.050928    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:27.215837 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.215803    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:27.383706 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.383677    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:27.462252 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.460537    6738 generic.go:356] "Generic (PLEG): container finished" podID="1304a3ec-0e83-4205-ab4b-be0d16fc7194" containerID="ba6d4d383adec02bdb23039f645dd72835b8b687751f97ab3e210bf9283b25de" exitCode=0
Oct 30 08:03:27.462252 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.460926    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n" event={"ID":"1304a3ec-0e83-4205-ab4b-be0d16fc7194","Type":"ContainerDied","Data":"ba6d4d383adec02bdb23039f645dd72835b8b687751f97ab3e210bf9283b25de"}
Oct 30 08:03:27.462252 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.460977    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n" event={"ID":"1304a3ec-0e83-4205-ab4b-be0d16fc7194","Type":"ContainerStarted","Data":"89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c"}
Oct 30 08:03:27.631617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.631587    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:27.744162 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.744134    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\") pod \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") "
Oct 30 08:03:27.744391 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.744377    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\" (UniqueName: \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\") pod \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") "
Oct 30 08:03:27.744472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.744465    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\") pod \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\" (UID: \"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d\") "
Oct 30 08:03:27.744525 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.744497    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca" pod "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" (UID: "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:27.744862 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.744840    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:27.746502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.746478    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca" pod "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" (UID: "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:27.746607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.746548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb" pod "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" (UID: "a9f2c037-b001-4ac8-8a1b-936e7f5fc59d"). InnerVolumeSpecName "kube-api-access-8hzzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:27.846386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.846344    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzzb\" (UniqueName: \"kubernetes.io/projected/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-kube-api-access-8hzzb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:27.846386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:27.846376    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:28.291382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.291342    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:28.470104 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.470072    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"
Oct 30 08:03:28.470339 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.470103    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8" event={"ID":"a9f2c037-b001-4ac8-8a1b-936e7f5fc59d","Type":"ContainerDied","Data":"cfb174cc506116f1b269d36a35c339f12e248fd45d5134711b43fb44c68e9b2f"}
Oct 30 08:03:28.470339 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.470169    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb174cc506116f1b269d36a35c339f12e248fd45d5134711b43fb44c68e9b2f"
Oct 30 08:03:28.633231 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.633204    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:28.756884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.756854    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\") pod \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") "
Oct 30 08:03:28.757068 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.756962    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\") pod \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") "
Oct 30 08:03:28.757146 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.757111    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\" (UniqueName: \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\") pod \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\" (UID: \"1304a3ec-0e83-4205-ab4b-be0d16fc7194\") "
Oct 30 08:03:28.757210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.757183    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca" pod "1304a3ec-0e83-4205-ab4b-be0d16fc7194" (UID: "1304a3ec-0e83-4205-ab4b-be0d16fc7194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:28.757535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.757513    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:28.759140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.759113    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca" pod "1304a3ec-0e83-4205-ab4b-be0d16fc7194" (UID: "1304a3ec-0e83-4205-ab4b-be0d16fc7194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:28.759255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.759164    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm" pod "1304a3ec-0e83-4205-ab4b-be0d16fc7194" (UID: "1304a3ec-0e83-4205-ab4b-be0d16fc7194"). InnerVolumeSpecName "kube-api-access-lqwtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:28.859379 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.859274    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1304a3ec-0e83-4205-ab4b-be0d16fc7194-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:28.859379 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.859322    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqwtm\" (UniqueName: \"kubernetes.io/projected/1304a3ec-0e83-4205-ab4b-be0d16fc7194-kube-api-access-lqwtm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:28.864458 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.864427    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:28.994503 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:28.994419    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:29.215068 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.215011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:29.348703 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.348668    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:29.463320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.461190    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 08:03:29.484318 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.480402    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484318 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484308    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:29.484592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484378    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484431    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484594    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484659    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484695    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.484800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484777    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.485220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484864    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.485220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.484917    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\") pod \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\" (UID: \"cdaeb338-56ba-43c4-aaee-36edb3fa3862\") "
Oct 30 08:03:29.488877 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.485551    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.488877 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.488112    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:29.493140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.490354    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:03:29.493140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.490711    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:29.503535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.494578    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "kube-api-access-h7mk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:29.503535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.494915    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:29.503535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.495117    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:29.503535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.495322    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:29.503535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.498931    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy" pod "cdaeb338-56ba-43c4-aaee-36edb3fa3862" (UID: "cdaeb338-56ba-43c4-aaee-36edb3fa3862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:29.514223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.514189    6738 generic.go:356] "Generic (PLEG): container finished" podID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerID="5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5" exitCode=0
Oct 30 08:03:29.514435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.514271    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"
Oct 30 08:03:29.514435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.514288    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerDied","Data":"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"}
Oct 30 08:03:29.514543 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.514440    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6" event={"ID":"cdaeb338-56ba-43c4-aaee-36edb3fa3862","Type":"ContainerDied","Data":"abf73941de5bba4f67ae679e0af830d7f35da7f4d6ecb1a97c9f93323d0a39e1"}
Oct 30 08:03:29.514543 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.514497    6738 scope.go:122] "RemoveContainer" containerID="5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"
Oct 30 08:03:29.516772 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.516729    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n" event={"ID":"1304a3ec-0e83-4205-ab4b-be0d16fc7194","Type":"ContainerDied","Data":"89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c"}
Oct 30 08:03:29.516772 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.516764    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"
Oct 30 08:03:29.516961 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.516834    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ac2c7f4581a9dda70bf9c0ed1f0530d486992c672e7d1ffe51c3b3b7859a7c"
Oct 30 08:03:29.528638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.528615    6738 scope.go:122] "RemoveContainer" containerID="24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"
Oct 30 08:03:29.539242 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.539219    6738 scope.go:122] "RemoveContainer" containerID="5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"
Oct 30 08:03:29.539615 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:29.539585    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5\": container with ID starting with 5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5 not found: ID does not exist" containerID="5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"
Oct 30 08:03:29.539686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.539616    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5"} err="failed to get container status \"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5\": rpc error: code = NotFound desc = could not find container \"5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5\": container with ID starting with 5ffe47dae08c54110b624c898b734af8a48ee10f3a76a18818aff7ecb7f9d4c5 not found: ID does not exist"
Oct 30 08:03:29.539686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.539636    6738 scope.go:122] "RemoveContainer" containerID="24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"
Oct 30 08:03:29.539921 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:29.539901    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26\": container with ID starting with 24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26 not found: ID does not exist" containerID="24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"
Oct 30 08:03:29.539988 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.539931    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26"} err="failed to get container status \"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26\": rpc error: code = NotFound desc = could not find container \"24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26\": container with ID starting with 24475ddba7d8fee3465790ffa9d0daf1ed9182774246e4b63c89d5e6607d0c26 not found: ID does not exist"
Oct 30 08:03:29.587097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587054    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mk5\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-kube-api-access-h7mk5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587092    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587108    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587116    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587124    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587133    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587140    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.587440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.587152    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdaeb338-56ba-43c4-aaee-36edb3fa3862-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:29.661534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.661445    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 08:03:29.668280 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.668250    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-932c00-openshift-default-76b46f778f-bd8q6"]
Oct 30 08:03:29.691728 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:29.691697    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" path="/var/lib/kubelet/pods/cdaeb338-56ba-43c4-aaee-36edb3fa3862/volumes"
Oct 30 08:03:30.677899 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:30.677866    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 08:03:30.693018 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:30.692986    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 08:03:30.707385 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:30.707356    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 08:03:30.788191 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:30.788155    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:31.332886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.332854    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:31.406410 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.405599    6738 scope.go:122] "RemoveContainer" containerID="8eed75af3e73bf30671702dc6004e2bea4e45b96a33ac541ac7edf53aa87aed7"
Oct 30 08:03:31.426167 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.426140    6738 scope.go:122] "RemoveContainer" containerID="88b88956743cd6a3aa8632ce75b86c288abec9f7e671c75be5960361fd5dfb20"
Oct 30 08:03:31.440887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.440860    6738 scope.go:122] "RemoveContainer" containerID="d813037fe2eb0e07b0ea22d3890d7510112c539ab59fcab6e2a4d65ab4a70852"
Oct 30 08:03:31.452084 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.452061    6738 scope.go:122] "RemoveContainer" containerID="a87e8fad2404ab059324a0a1cb7dc3b159b27d7a34e20768696829a3e3b13d58"
Oct 30 08:03:31.463589 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:31.463567    6738 scope.go:122] "RemoveContainer" containerID="b4c265cf171698e44d3147755fbb1b2a8cb2e3ee53fa81b47869703579dcff2e"
Oct 30 08:03:33.119583 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:33.119548    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:34.341015 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.340944    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 08:03:34.451598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.451564    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\" (UniqueName: \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\") pod \"e7b181a4-57f2-4897-a695-4aaf837e1647\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") "
Oct 30 08:03:34.451765 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.451634    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\") pod \"e7b181a4-57f2-4897-a695-4aaf837e1647\" (UID: \"e7b181a4-57f2-4897-a695-4aaf837e1647\") "
Oct 30 08:03:34.452066 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.452038    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume" pod "e7b181a4-57f2-4897-a695-4aaf837e1647" (UID: "e7b181a4-57f2-4897-a695-4aaf837e1647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:34.453788 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.453765    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95" pod "e7b181a4-57f2-4897-a695-4aaf837e1647" (UID: "e7b181a4-57f2-4897-a695-4aaf837e1647"). InnerVolumeSpecName "kube-api-access-6fm95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:34.553819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.553775    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fm95\" (UniqueName: \"kubernetes.io/projected/e7b181a4-57f2-4897-a695-4aaf837e1647-kube-api-access-6fm95\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:34.553819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.553819    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b181a4-57f2-4897-a695-4aaf837e1647-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:34.563510 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.563473    6738 generic.go:356] "Generic (PLEG): container finished" podID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerID="d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6" exitCode=0
Oct 30 08:03:34.563716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.563526    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"
Oct 30 08:03:34.563716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.563564    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" event={"ID":"e7b181a4-57f2-4897-a695-4aaf837e1647","Type":"ContainerDied","Data":"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"}
Oct 30 08:03:34.563716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.563638    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77" event={"ID":"e7b181a4-57f2-4897-a695-4aaf837e1647","Type":"ContainerDied","Data":"98ed1235984a9868afe0cb3ab7ae412085d622cacd0c032ff6e3264c60fee750"}
Oct 30 08:03:34.563716 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.563673    6738 scope.go:122] "RemoveContainer" containerID="d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"
Oct 30 08:03:34.588603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.578436    6738 scope.go:122] "RemoveContainer" containerID="d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"
Oct 30 08:03:34.588603 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:34.583808    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6\": container with ID starting with d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6 not found: ID does not exist" containerID="d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"
Oct 30 08:03:34.588603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.583848    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6"} err="failed to get container status \"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6\": rpc error: code = NotFound desc = could not find container \"d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6\": container with ID starting with d10489fc898ca69f17d21a7848ae8d7703c5ef153b8d624535e2d107ede09df6 not found: ID does not exist"
Oct 30 08:03:34.618491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.618395    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 08:03:34.627120 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.627081    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-932c00-68ddcd6bdb-82k77"]
Oct 30 08:03:34.925699 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:34.925659    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:35.380444 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:35.380410    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:35.519315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:35.518999    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:35.693237 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:35.693154    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" path="/var/lib/kubelet/pods/e7b181a4-57f2-4897-a695-4aaf837e1647/volumes"
Oct 30 08:03:36.039949 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.039855    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:36.384497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.384453    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 08:03:36.419369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.419334    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 08:03:36.538097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.538060    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:36.656822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.656721    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:36.965217 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:36.965120    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:37.520278 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:37.520244    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:03:38.834559 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:38.834520    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:40.585949 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:40.585858    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:41.385771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:41.385725    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing" containerID="cri-o://7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249" gracePeriod=25
Oct 30 08:03:41.419935 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:41.419865    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing" containerID="cri-o://e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8" gracePeriod=25
Oct 30 08:03:42.375558 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:42.375521    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:42.646143 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:42.646054    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:44.402199 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:44.402158    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:44.662177 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:44.662081    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:45.700628 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:45.700592    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" containerID="cri-o://ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e" gracePeriod=45
Oct 30 08:03:45.720870 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:45.720840    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" containerID="cri-o://95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea" gracePeriod=45
Oct 30 08:03:45.770023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:45.769982    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused"
Oct 30 08:03:45.835864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:45.835825    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": dial tcp 10.133.0.59:8000: connect: connection refused"
Oct 30 08:03:45.985428 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:45.985398    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 08:03:46.024757 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.024716    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 08:03:46.163576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163541    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.163761 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163594    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.163761 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163622    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\" (UniqueName: \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.163761 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163683    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.163938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163815    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.163938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163831    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.163938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163888    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.163938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163888    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.164193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.163947    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.164193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164035    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.164193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164054    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.164193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164146    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.164193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164185    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.164612 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164204    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.164612 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164223    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\" (UniqueName: \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\") pod \"5dcfdfac-7675-4576-835a-373478897a9e\" (UID: \"5dcfdfac-7675-4576-835a-373478897a9e\") "
Oct 30 08:03:46.164612 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164266    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.164612 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164355    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\") pod \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\" (UID: \"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff\") "
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164966    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.164991    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.165004    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.165020    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.165033    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.165226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.165035    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.166570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.166512    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.166570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.166538    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "kube-api-access-jlcc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:46.166762 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.166566    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "kube-api-access-8mqrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:46.166762 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.166600    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:46.166762 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.166668    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs" pod "5dcfdfac-7675-4576-835a-373478897a9e" (UID: "5dcfdfac-7675-4576-835a-373478897a9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:46.167182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.167167    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm" pod "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" (UID: "7ff76749-6812-40a4-aa07-9d3ecb6ba0ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.265968    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dcfdfac-7675-4576-835a-373478897a9e-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266011    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcfdfac-7675-4576-835a-373478897a9e-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266022    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266030    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqrl\" (UniqueName: \"kubernetes.io/projected/5dcfdfac-7675-4576-835a-373478897a9e-kube-api-access-8mqrl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266040    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266049    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcc9\" (UniqueName: \"kubernetes.io/projected/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-kube-api-access-jlcc9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.266072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.266057    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.366056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.366015    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:46.554063 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.553968    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main" containerID="cri-o://639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375" gracePeriod=45
Oct 30 08:03:46.652340 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.652270    6738 generic.go:356] "Generic (PLEG): container finished" podID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerID="95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea" exitCode=0
Oct 30 08:03:46.652549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.652353    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"
Oct 30 08:03:46.652549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.652354    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" event={"ID":"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff","Type":"ContainerDied","Data":"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"}
Oct 30 08:03:46.652549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.652403    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n" event={"ID":"7ff76749-6812-40a4-aa07-9d3ecb6ba0ff","Type":"ContainerDied","Data":"746ba0ed0e3dc0b7b127b3a598d4b985e2d64bc49cda19e2980eb5f33c1c4733"}
Oct 30 08:03:46.652549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.652423    6738 scope.go:122] "RemoveContainer" containerID="95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"
Oct 30 08:03:46.653862 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.653839    6738 generic.go:356] "Generic (PLEG): container finished" podID="5dcfdfac-7675-4576-835a-373478897a9e" containerID="ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e" exitCode=0
Oct 30 08:03:46.653944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.653875    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"
Oct 30 08:03:46.653944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.653877    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" event={"ID":"5dcfdfac-7675-4576-835a-373478897a9e","Type":"ContainerDied","Data":"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"}
Oct 30 08:03:46.653944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.653908    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g" event={"ID":"5dcfdfac-7675-4576-835a-373478897a9e","Type":"ContainerDied","Data":"ca2ed8ad972e1eeb8481714980f913f8bd023d5019d7a97c963c24f0e8fa7f32"}
Oct 30 08:03:46.667969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.667935    6738 scope.go:122] "RemoveContainer" containerID="95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"
Oct 30 08:03:46.668399 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:46.668374    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea\": container with ID starting with 95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea not found: ID does not exist" containerID="95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"
Oct 30 08:03:46.668515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.668407    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea"} err="failed to get container status \"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea\": rpc error: code = NotFound desc = could not find container \"95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea\": container with ID starting with 95c686d0f2819f8aa3cb46e18093d2306bbcb191f4dc62c7e9e75b91a61c55ea not found: ID does not exist"
Oct 30 08:03:46.668515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.668424    6738 scope.go:122] "RemoveContainer" containerID="ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"
Oct 30 08:03:46.679451 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.679431    6738 scope.go:122] "RemoveContainer" containerID="ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"
Oct 30 08:03:46.679807 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:46.679780    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e\": container with ID starting with ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e not found: ID does not exist" containerID="ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"
Oct 30 08:03:46.679889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.679823    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e"} err="failed to get container status \"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e\": rpc error: code = NotFound desc = could not find container \"ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e\": container with ID starting with ad1e8c6c8b508c0718d62f6073681bdae7fd227265e588afe5bd86b945278d0e not found: ID does not exist"
Oct 30 08:03:46.707760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.707726    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 08:03:46.721996 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.721862    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/distinct-2-w1-4fe2ed-kserve-78f6fccc67-lct8g"]
Oct 30 08:03:46.748722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.748690    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 08:03:46.752530 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.752502    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/premium-w1-4fe2ed-kserve-74b54c5f49-kgl8n"]
Oct 30 08:03:46.832274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.832244    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 08:03:46.973894 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.973856    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.973925    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\" (UniqueName: \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.973972    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974073    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974245 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974145    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.974245 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974198    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974245 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974237    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\") pod \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\" (UID: \"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed\") "
Oct 30 08:03:46.974432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974243    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.974490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974423    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:46.974835 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974808    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.974947 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974839    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.974947 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.974879    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:46.976258 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.976236    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "kube-api-access-9gkt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:46.976376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.976321    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:46.976445 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:46.976407    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm" pod "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" (UID: "e0b4fd7e-470b-424c-8a3f-79f7b2d302ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:03:47.076115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.076030    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:47.076115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.076063    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkt4\" (UniqueName: \"kubernetes.io/projected/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-kube-api-access-9gkt4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:47.076115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.076072    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:47.665750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.665712    6738 generic.go:356] "Generic (PLEG): container finished" podID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerID="639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375" exitCode=0
Oct 30 08:03:47.665966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.665757    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"
Oct 30 08:03:47.665966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.665780    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" event={"ID":"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed","Type":"ContainerDied","Data":"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"}
Oct 30 08:03:47.665966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.665812    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf" event={"ID":"e0b4fd7e-470b-424c-8a3f-79f7b2d302ed","Type":"ContainerDied","Data":"689dece6e6ee41cb839bc15f319a7c8dc84495f8d60530c79b34153da34bf396"}
Oct 30 08:03:47.665966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.665828    6738 scope.go:122] "RemoveContainer" containerID="639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"
Oct 30 08:03:47.677115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.677088    6738 scope.go:122] "RemoveContainer" containerID="639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"
Oct 30 08:03:47.677478 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:03:47.677454    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375\": container with ID starting with 639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375 not found: ID does not exist" containerID="639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"
Oct 30 08:03:47.677561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.677490    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375"} err="failed to get container status \"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375\": rpc error: code = NotFound desc = could not find container \"639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375\": container with ID starting with 639cd344f732b682a6506b53a1e10ce90596b7851061da1630a75f9aca6a2375 not found: ID does not exist"
Oct 30 08:03:47.689609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.689579    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcfdfac-7675-4576-835a-373478897a9e" path="/var/lib/kubelet/pods/5dcfdfac-7675-4576-835a-373478897a9e/volumes"
Oct 30 08:03:47.690045 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.690032    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" path="/var/lib/kubelet/pods/7ff76749-6812-40a4-aa07-9d3ecb6ba0ff/volumes"
Oct 30 08:03:47.741130 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.741095    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 08:03:47.758817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:47.758778    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe2ed/simulator-w1-4fe2ed-kserve-55f568947b-gd4mf"]
Oct 30 08:03:48.096794 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.096690    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:48.634332 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.604231    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"]
Oct 30 08:03:48.650931 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.636000    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main"
Oct 30 08:03:48.651169 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651143    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main"
Oct 30 08:03:48.651325 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651312    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.651447 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651435    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.651589 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651577    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1304a3ec-0e83-4205-ab4b-be0d16fc7194" containerName="revoke-keys"
Oct 30 08:03:48.651685 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651672    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1304a3ec-0e83-4205-ab4b-be0d16fc7194" containerName="revoke-keys"
Oct 30 08:03:48.651780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651768    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main"
Oct 30 08:03:48.651878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651865    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main"
Oct 30 08:03:48.651974 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.651964    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing"
Oct 30 08:03:48.652043 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652036    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing"
Oct 30 08:03:48.652110 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652104    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.652157 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652151    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.652228 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652221    6738 cpu_manager.go:414] "containerMap: removing container" podUID="a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" containerName="revoke-keys"
Oct 30 08:03:48.652286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652279    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" containerName="revoke-keys"
Oct 30 08:03:48.652382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652375    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main"
Oct 30 08:03:48.652439 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652432    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main"
Oct 30 08:03:48.652626 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652618    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" containerName="main"
Oct 30 08:03:48.652685 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652678    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.652741 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652735    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1304a3ec-0e83-4205-ab4b-be0d16fc7194" containerName="revoke-keys"
Oct 30 08:03:48.652809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652802    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcfdfac-7675-4576-835a-373478897a9e" containerName="main"
Oct 30 08:03:48.652865 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652859    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b181a4-57f2-4897-a695-4aaf837e1647" containerName="payload-processing"
Oct 30 08:03:48.652924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652918    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" containerName="revoke-keys"
Oct 30 08:03:48.652976 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.652971    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff76749-6812-40a4-aa07-9d3ecb6ba0ff" containerName="main"
Oct 30 08:03:48.658280 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.658253    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"]
Oct 30 08:03:48.658447 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.658432    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.659626 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.659596    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdaeb338-56ba-43c4-aaee-36edb3fa3862" containerName="istio-proxy"
Oct 30 08:03:48.663636 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.663613    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"]
Oct 30 08:03:48.663737 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.663658    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"]
Oct 30 08:03:48.663791 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.663748    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.676647 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.674798    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:48.676647 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.675067    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:03:48.676647 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.675187    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:03:48.805008 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.804966    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt6z\" (UniqueName: \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.805527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.805029    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.805527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.805208    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.805527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.805244    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.805527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.805326    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.805527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.805383    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw87\" (UniqueName: \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.888496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.888412    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:48.907101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907062    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.907353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907169    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.907353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907204    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.907353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907248    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.907353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907326    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw87\" (UniqueName: \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.907629 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907382    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt6z\" (UniqueName: \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.907930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907902    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.908054 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.907981    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.909592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.909571    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.909709 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.909641    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.947163 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.945978    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt6z\" (UniqueName: \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\") pod \"maas-api-revoke-sub-ab3ca735-jlvcz\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:48.949965 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.949521    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw87\" (UniqueName: \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\") pod \"maas-api-revoke-sub-4d5565d0-7nxtl\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.950808 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.950777    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"]
Oct 30 08:03:48.954248 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.954220    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-137122-tkkt8"]
Oct 30 08:03:48.978726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.978696    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:48.987555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:48.987528    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:49.094812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.094627    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:49.309926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.309886    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"]
Oct 30 08:03:49.310631 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:03:49.310604    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6361e7e_4c50_47ef_8ca9_6ca85adce8fb.slice/crio-30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4 WatchSource:0}: Error finding container 30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4: Status 404 returned error can't find the container with id 30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4
Oct 30 08:03:49.320848 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.320823    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"]
Oct 30 08:03:49.324271 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:03:49.324244    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffab17de_3d1b_40cb_b795_48fa6c546c5d.slice/crio-e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5 WatchSource:0}: Error finding container e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5: Status 404 returned error can't find the container with id e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5
Oct 30 08:03:49.497059 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.497028    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:49.696592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.696560    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f2c037-b001-4ac8-8a1b-936e7f5fc59d" path="/var/lib/kubelet/pods/a9f2c037-b001-4ac8-8a1b-936e7f5fc59d/volumes"
Oct 30 08:03:49.697209 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.697183    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b4fd7e-470b-424c-8a3f-79f7b2d302ed" path="/var/lib/kubelet/pods/e0b4fd7e-470b-424c-8a3f-79f7b2d302ed/volumes"
Oct 30 08:03:49.697535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.697516    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl" event={"ID":"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb","Type":"ContainerStarted","Data":"5b7776a21182ccb4bb210bfc82668d117ae5dcd740d4a1fe9e84ab8479680011"}
Oct 30 08:03:49.697586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.697560    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl" event={"ID":"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb","Type":"ContainerStarted","Data":"30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4"}
Oct 30 08:03:49.697586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.697577    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz" event={"ID":"ffab17de-3d1b-40cb-b795-48fa6c546c5d","Type":"ContainerStarted","Data":"c733771563895bcee146ed978114568e8e7f011c1f281d34310d0da4874ca862"}
Oct 30 08:03:49.697672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.697598    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz" event={"ID":"ffab17de-3d1b-40cb-b795-48fa6c546c5d","Type":"ContainerStarted","Data":"e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5"}
Oct 30 08:03:49.740537 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.740473    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl" podStartSLOduration=1.740454065 podStartE2EDuration="1.740454065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:49.737574641 +0000 UTC m=+3990.170338534" watchObservedRunningTime="2026-10-30 08:03:49.740454065 +0000 UTC m=+3990.173217959"
Oct 30 08:03:49.782190 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:49.782117    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz" podStartSLOduration=1.782099236 podStartE2EDuration="1.782099236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:49.779149929 +0000 UTC m=+3990.211913821" watchObservedRunningTime="2026-10-30 08:03:49.782099236 +0000 UTC m=+3990.214863166"
Oct 30 08:03:50.708638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:50.708600    6738 generic.go:356] "Generic (PLEG): container finished" podID="d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" containerID="5b7776a21182ccb4bb210bfc82668d117ae5dcd740d4a1fe9e84ab8479680011" exitCode=0
Oct 30 08:03:50.709146 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:50.708679    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl" event={"ID":"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb","Type":"ContainerDied","Data":"5b7776a21182ccb4bb210bfc82668d117ae5dcd740d4a1fe9e84ab8479680011"}
Oct 30 08:03:50.710440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:50.710410    6738 generic.go:356] "Generic (PLEG): container finished" podID="ffab17de-3d1b-40cb-b795-48fa6c546c5d" containerID="c733771563895bcee146ed978114568e8e7f011c1f281d34310d0da4874ca862" exitCode=0
Oct 30 08:03:50.710592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:50.710495    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz" event={"ID":"ffab17de-3d1b-40cb-b795-48fa6c546c5d","Type":"ContainerDied","Data":"c733771563895bcee146ed978114568e8e7f011c1f281d34310d0da4874ca862"}
Oct 30 08:03:51.206223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.206184    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:51.877272 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.877238    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:51.910167 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.910139    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:51.975019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.974934    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\") pod \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") "
Oct 30 08:03:51.975019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.974984    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\" (UniqueName: \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\") pod \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") "
Oct 30 08:03:51.975281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975046    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\") pod \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") "
Oct 30 08:03:51.975281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975091    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\" (UniqueName: \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\") pod \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") "
Oct 30 08:03:51.975281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975227    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\") pod \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\" (UID: \"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb\") "
Oct 30 08:03:51.975526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975385    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca" pod "ffab17de-3d1b-40cb-b795-48fa6c546c5d" (UID: "ffab17de-3d1b-40cb-b795-48fa6c546c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:51.975526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975484    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\") pod \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\" (UID: \"ffab17de-3d1b-40cb-b795-48fa6c546c5d\") "
Oct 30 08:03:51.975773 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.975746    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca" pod "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" (UID: "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:51.976275 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.976252    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:51.976456 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.976429    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:51.978028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.977999    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca" pod "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" (UID: "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:51.978391 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.978365    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87" pod "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" (UID: "d6361e7e-4c50-47ef-8ca9-6ca85adce8fb"). InnerVolumeSpecName "kube-api-access-cqw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:51.978641 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.978610    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z" pod "ffab17de-3d1b-40cb-b795-48fa6c546c5d" (UID: "ffab17de-3d1b-40cb-b795-48fa6c546c5d"). InnerVolumeSpecName "kube-api-access-flt6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:51.978749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:51.978653    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca" pod "ffab17de-3d1b-40cb-b795-48fa6c546c5d" (UID: "ffab17de-3d1b-40cb-b795-48fa6c546c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:03:52.077655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.077617    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffab17de-3d1b-40cb-b795-48fa6c546c5d-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:52.077655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.077649    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw87\" (UniqueName: \"kubernetes.io/projected/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-kube-api-access-cqw87\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:52.077655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.077661    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:52.077930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.077669    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flt6z\" (UniqueName: \"kubernetes.io/projected/ffab17de-3d1b-40cb-b795-48fa6c546c5d-kube-api-access-flt6z\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:03:52.646115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.646073    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:52.729791 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.729758    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"
Oct 30 08:03:52.730014 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.729794    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl" event={"ID":"d6361e7e-4c50-47ef-8ca9-6ca85adce8fb","Type":"ContainerDied","Data":"30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4"}
Oct 30 08:03:52.730014 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.729849    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a3d381d97097a413a637613912c89581c8652ba324f80f6eec17cee1e782b4"
Oct 30 08:03:52.731467 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.731446    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"
Oct 30 08:03:52.731638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.731485    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz" event={"ID":"ffab17de-3d1b-40cb-b795-48fa6c546c5d","Type":"ContainerDied","Data":"e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5"}
Oct 30 08:03:52.731638 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:52.731525    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e875eb730fa5088afde7731a542d94f63423e6b79a9120ab903205fc4cd8a6a5"
Oct 30 08:03:53.042233 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:53.042137    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:54.662908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:54.662864    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:54.942149 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:54.942054    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:56.203270 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.203234    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:56.315045 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.315012    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:56.535387 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.535285    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 08:03:56.535866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.535821    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy" containerID="cri-o://0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd" gracePeriod=30
Oct 30 08:03:56.575420 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.575386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:56.798917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:56.798822    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:57.070726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.070623    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:57.287419 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.287382    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"]
Oct 30 08:03:57.288214 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288190    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" containerName="revoke-keys"
Oct 30 08:03:57.288359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288216    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" containerName="revoke-keys"
Oct 30 08:03:57.288359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288338    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ffab17de-3d1b-40cb-b795-48fa6c546c5d" containerName="revoke-keys"
Oct 30 08:03:57.288359 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288348    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffab17de-3d1b-40cb-b795-48fa6c546c5d" containerName="revoke-keys"
Oct 30 08:03:57.288546 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288497    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffab17de-3d1b-40cb-b795-48fa6c546c5d" containerName="revoke-keys"
Oct 30 08:03:57.288546 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.288513    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" containerName="revoke-keys"
Oct 30 08:03:57.291995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.291972    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.310823 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.310768    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"]
Oct 30 08:03:57.362520 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.362430    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.362520 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.362484    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.362726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.362602    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnnv\" (UniqueName: \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.397678 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.397637    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:03:57.397963 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.397944    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:03:57.464056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.464012    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.464243 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.464116    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.464243 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.464234    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnnv\" (UniqueName: \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.465157 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.465132    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.466801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.466775    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.489352 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.488954    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnnv\" (UniqueName: \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.607827 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.607794    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:03:57.911346 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:03:57.911308    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d842e4_9d28_4f35_a87c_3c7c283de05c.slice/crio-7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573 WatchSource:0}: Error finding container 7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573: Status 404 returned error can't find the container with id 7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573
Oct 30 08:03:57.911579 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:57.911555    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"]
Oct 30 08:03:58.787554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:58.787518    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp" event={"ID":"c4d842e4-9d28-4f35-a87c-3c7c283de05c","Type":"ContainerStarted","Data":"0e85eb88369881f8058c45cf0cdec07dbe5a0938a77349ba285ce262aa711257"}
Oct 30 08:03:58.788026 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:58.787574    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp" event={"ID":"c4d842e4-9d28-4f35-a87c-3c7c283de05c","Type":"ContainerStarted","Data":"7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573"}
Oct 30 08:03:58.829876 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:58.829822    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp" podStartSLOduration=1.8298050639999999 podStartE2EDuration="1.829805064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:58.828129175 +0000 UTC m=+3999.260893068" watchObservedRunningTime="2026-10-30 08:03:58.829805064 +0000 UTC m=+3999.262568957"
Oct 30 08:03:58.879935 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:58.879891    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:03:59.797685 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:59.797650    6738 generic.go:356] "Generic (PLEG): container finished" podID="c4d842e4-9d28-4f35-a87c-3c7c283de05c" containerID="0e85eb88369881f8058c45cf0cdec07dbe5a0938a77349ba285ce262aa711257" exitCode=0
Oct 30 08:03:59.798107 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:03:59.797713    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp" event={"ID":"c4d842e4-9d28-4f35-a87c-3c7c283de05c","Type":"ContainerDied","Data":"0e85eb88369881f8058c45cf0cdec07dbe5a0938a77349ba285ce262aa711257"}
Oct 30 08:04:00.741882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:00.741845    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:00.966013 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:00.965971    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:04:01.007790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.007686    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\") pod \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") "
Oct 30 08:04:01.007790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.007787    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\" (UniqueName: \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\") pod \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") "
Oct 30 08:04:01.008065 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.007997    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\") pod \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\" (UID: \"c4d842e4-9d28-4f35-a87c-3c7c283de05c\") "
Oct 30 08:04:01.008591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.008560    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca" pod "c4d842e4-9d28-4f35-a87c-3c7c283de05c" (UID: "c4d842e4-9d28-4f35-a87c-3c7c283de05c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:01.011579 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.011533    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca" pod "c4d842e4-9d28-4f35-a87c-3c7c283de05c" (UID: "c4d842e4-9d28-4f35-a87c-3c7c283de05c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:01.011750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.011722    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv" pod "c4d842e4-9d28-4f35-a87c-3c7c283de05c" (UID: "c4d842e4-9d28-4f35-a87c-3c7c283de05c"). InnerVolumeSpecName "kube-api-access-4tnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:01.109648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.109603    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.109648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.109640    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d842e4-9d28-4f35-a87c-3c7c283de05c-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.109648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.109654    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnnv\" (UniqueName: \"kubernetes.io/projected/c4d842e4-9d28-4f35-a87c-3c7c283de05c-kube-api-access-4tnnv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.681399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.681365    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 08:04:01.718697 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.718662    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 08:04:01.808961 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.808933    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 08:04:01.817944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.817912    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\") pod \"3264b4cb-dde2-45f2-805b-dca581cdfe56\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") "
Oct 30 08:04:01.818109 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.817951    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\" (UniqueName: \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\") pod \"3264b4cb-dde2-45f2-805b-dca581cdfe56\" (UID: \"3264b4cb-dde2-45f2-805b-dca581cdfe56\") "
Oct 30 08:04:01.818109 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818077    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\" (UniqueName: \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\") pod \"ca930885-68e5-4124-a913-1550807c0c0b\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") "
Oct 30 08:04:01.818257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818108    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\") pod \"ca930885-68e5-4124-a913-1550807c0c0b\" (UID: \"ca930885-68e5-4124-a913-1550807c0c0b\") "
Oct 30 08:04:01.818356 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818315    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume" pod "3264b4cb-dde2-45f2-805b-dca581cdfe56" (UID: "3264b4cb-dde2-45f2-805b-dca581cdfe56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:01.818634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818605    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume" pod "ca930885-68e5-4124-a913-1550807c0c0b" (UID: "ca930885-68e5-4124-a913-1550807c0c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:01.818770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818718    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3264b4cb-dde2-45f2-805b-dca581cdfe56-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.818770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.818742    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca930885-68e5-4124-a913-1550807c0c0b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.820272 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.820246    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs" pod "ca930885-68e5-4124-a913-1550807c0c0b" (UID: "ca930885-68e5-4124-a913-1550807c0c0b"). InnerVolumeSpecName "kube-api-access-wszhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:01.820394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.820283    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh" pod "3264b4cb-dde2-45f2-805b-dca581cdfe56" (UID: "3264b4cb-dde2-45f2-805b-dca581cdfe56"). InnerVolumeSpecName "kube-api-access-726jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:01.824402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.824378    6738 generic.go:356] "Generic (PLEG): container finished" podID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerID="e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8" exitCode=0
Oct 30 08:04:01.824509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.824438    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"
Oct 30 08:04:01.824509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.824457    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" event={"ID":"3264b4cb-dde2-45f2-805b-dca581cdfe56","Type":"ContainerDied","Data":"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"}
Oct 30 08:04:01.824615 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.824512    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch" event={"ID":"3264b4cb-dde2-45f2-805b-dca581cdfe56","Type":"ContainerDied","Data":"af324aac0de14fb74c6b352c78b0da4d622841b1bccaf1ef483c687e7d343470"}
Oct 30 08:04:01.824615 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.824541    6738 scope.go:122] "RemoveContainer" containerID="e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"
Oct 30 08:04:01.827917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.827884    6738 generic.go:356] "Generic (PLEG): container finished" podID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerID="0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd" exitCode=0
Oct 30 08:04:01.828053 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.828034    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerDied","Data":"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"}
Oct 30 08:04:01.828135 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.828124    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f" event={"ID":"168c87eb-15fb-4103-bbcb-8ca59a136d09","Type":"ContainerDied","Data":"64adff5d7b35302a8f8975fad3600ca32493b4879bb23de6c8131f129ba00eb2"}
Oct 30 08:04:01.828279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.828260    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"
Oct 30 08:04:01.830288 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.830251    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp" event={"ID":"c4d842e4-9d28-4f35-a87c-3c7c283de05c","Type":"ContainerDied","Data":"7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573"}
Oct 30 08:04:01.830408 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.830330    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e353ea31e7bde0d707533c6ff625aa551ce4a5d83d6cdf571d1afefaf1ef573"
Oct 30 08:04:01.830467 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.830437    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"
Oct 30 08:04:01.832161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.832128    6738 generic.go:356] "Generic (PLEG): container finished" podID="ca930885-68e5-4124-a913-1550807c0c0b" containerID="7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249" exitCode=0
Oct 30 08:04:01.832264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.832228    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" event={"ID":"ca930885-68e5-4124-a913-1550807c0c0b","Type":"ContainerDied","Data":"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"}
Oct 30 08:04:01.832355 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.832279    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk" event={"ID":"ca930885-68e5-4124-a913-1550807c0c0b","Type":"ContainerDied","Data":"c363ada19ee05e3b21cb4fe9d0e03374ee6d5be74a3c9fa46f81040a4df09117"}
Oct 30 08:04:01.832418 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.832386    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"
Oct 30 08:04:01.850565 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.850540    6738 scope.go:122] "RemoveContainer" containerID="e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"
Oct 30 08:04:01.850938 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:01.850915    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8\": container with ID starting with e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8 not found: ID does not exist" containerID="e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"
Oct 30 08:04:01.851044 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.850947    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8"} err="failed to get container status \"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8\": rpc error: code = NotFound desc = could not find container \"e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8\": container with ID starting with e259112bcb7429f47c3235f0703f8e09d0ef0c08c1c049e505758749babe58b8 not found: ID does not exist"
Oct 30 08:04:01.851044 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.850968    6738 scope.go:122] "RemoveContainer" containerID="0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"
Oct 30 08:04:01.871050 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.871017    6738 scope.go:122] "RemoveContainer" containerID="dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"
Oct 30 08:04:01.885341 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.885275    6738 scope.go:122] "RemoveContainer" containerID="0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"
Oct 30 08:04:01.885742 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:01.885716    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd\": container with ID starting with 0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd not found: ID does not exist" containerID="0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"
Oct 30 08:04:01.885860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.885747    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd"} err="failed to get container status \"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd\": rpc error: code = NotFound desc = could not find container \"0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd\": container with ID starting with 0496914a7b2a4d59df1b4d4adf0466a4a94191e1ee88eda5d56399b30bf9e3dd not found: ID does not exist"
Oct 30 08:04:01.885860 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.885767    6738 scope.go:122] "RemoveContainer" containerID="dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"
Oct 30 08:04:01.886103 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:01.886079    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6\": container with ID starting with dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6 not found: ID does not exist" containerID="dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"
Oct 30 08:04:01.886176 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.886114    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6"} err="failed to get container status \"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6\": rpc error: code = NotFound desc = could not find container \"dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6\": container with ID starting with dfef1e99ebdbfe0df88e285cecd73ed5a938ef9561ff14f188c6c9d99f9ab3d6 not found: ID does not exist"
Oct 30 08:04:01.886176 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.886139    6738 scope.go:122] "RemoveContainer" containerID="7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"
Oct 30 08:04:01.901193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.898559    6738 scope.go:122] "RemoveContainer" containerID="7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"
Oct 30 08:04:01.901193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.899333    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 08:04:01.902075 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:01.902038    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249\": container with ID starting with 7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249 not found: ID does not exist" containerID="7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"
Oct 30 08:04:01.902224 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.902081    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249"} err="failed to get container status \"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249\": rpc error: code = NotFound desc = could not find container \"7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249\": container with ID starting with 7e38a0c8f971993876ea93236e4af83452fa4a2ff1aaa6d9d8e03398e6386249 not found: ID does not exist"
Oct 30 08:04:01.902746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.902724    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe2ed-5d8b745f9b-5qfrk"]
Oct 30 08:04:01.919555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919464    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919538    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919776 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919569    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919776 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919738    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919868 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919773    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:01.919868 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919841    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919878    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.919950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919917    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.920057 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919965    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:01.920057 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919978    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:01.920057 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.919984    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.920224 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920085    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\") pod \"168c87eb-15fb-4103-bbcb-8ca59a136d09\" (UID: \"168c87eb-15fb-4103-bbcb-8ca59a136d09\") "
Oct 30 08:04:01.920224 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920197    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:01.920389 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920359    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:01.920704 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920684    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920704 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920704    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920714    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c87eb-15fb-4103-bbcb-8ca59a136d09-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920724    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszhs\" (UniqueName: \"kubernetes.io/projected/ca930885-68e5-4124-a913-1550807c0c0b-kube-api-access-wszhs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920734    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920742    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.920878 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.920758    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-726jh\" (UniqueName: \"kubernetes.io/projected/3264b4cb-dde2-45f2-805b-dca581cdfe56-kube-api-access-726jh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.922029 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.922003    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:04:01.922151 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.922085    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:01.922324 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.922277    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "kube-api-access-srbvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:01.922413 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.922394    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy" pod "168c87eb-15fb-4103-bbcb-8ca59a136d09" (UID: "168c87eb-15fb-4103-bbcb-8ca59a136d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:01.937074 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.937043    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 08:04:01.940881 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.940850    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe2ed-5d64d7898c-xx7ch"]
Oct 30 08:04:01.985349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:01.985312    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 08:04:02.022542 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.022504    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:02.022542 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.022534    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbvh\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-kube-api-access-srbvh\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:02.022542 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.022545    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:02.022801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.022555    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c87eb-15fb-4103-bbcb-8ca59a136d09-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:02.110713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.110675    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:02.181699 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.181611    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 08:04:02.188158 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.188125    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-137122-openshift-default-74649f747c-jcf8f"]
Oct 30 08:04:02.549100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:02.549008    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:03.689982 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:03.689948    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" path="/var/lib/kubelet/pods/168c87eb-15fb-4103-bbcb-8ca59a136d09/volumes"
Oct 30 08:04:03.690426 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:03.690416    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" path="/var/lib/kubelet/pods/3264b4cb-dde2-45f2-805b-dca581cdfe56/volumes"
Oct 30 08:04:03.690798 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:03.690784    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca930885-68e5-4124-a913-1550807c0c0b" path="/var/lib/kubelet/pods/ca930885-68e5-4124-a913-1550807c0c0b/volumes"
Oct 30 08:04:04.446777 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:04.446737    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:06.342001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:06.341502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:07.522817 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:07.522775    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:04:07.979611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:07.979571    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:08.116267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:08.115416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:08.442937 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:08.442887    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:08.968358 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:08.968289    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:09.285902 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:09.285818    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:09.552419 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:09.552329    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:09.882028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:09.881995    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:10.212591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.212502    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"]
Oct 30 08:04:10.213401 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213382    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213402    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213413    6738 cpu_manager.go:414] "containerMap: removing container" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213418    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213455    6738 cpu_manager.go:414] "containerMap: removing container" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213461    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213478    6738 cpu_manager.go:414] "containerMap: removing container" podUID="c4d842e4-9d28-4f35-a87c-3c7c283de05c" containerName="revoke-keys"
Oct 30 08:04:10.213482 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213482    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d842e4-9d28-4f35-a87c-3c7c283de05c" containerName="revoke-keys"
Oct 30 08:04:10.213725 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213563    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d842e4-9d28-4f35-a87c-3c7c283de05c" containerName="revoke-keys"
Oct 30 08:04:10.213725 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213571    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca930885-68e5-4124-a913-1550807c0c0b" containerName="payload-processing"
Oct 30 08:04:10.213725 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213577    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.213725 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.213590    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="3264b4cb-dde2-45f2-805b-dca581cdfe56" containerName="payload-pre-processing"
Oct 30 08:04:10.219062 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.219038    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.234515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.234480    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"]
Oct 30 08:04:10.243577 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.243546    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:10.248340 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.248285    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:10.251622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.251592    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"]
Oct 30 08:04:10.252227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.252206    6738 cpu_manager.go:414] "containerMap: removing container" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.252227 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.252222    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.252487 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.252395    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="168c87eb-15fb-4103-bbcb-8ca59a136d09" containerName="istio-proxy"
Oct 30 08:04:10.255926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.255906    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.262137 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.262106    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.264723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.264691    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgq8\" (UniqueName: \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.265021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.264992    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.265156 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.265043    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.265156 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.265118    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjkf\" (UniqueName: \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.265336 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.265183    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.279240 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.279202    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"]
Oct 30 08:04:10.366982 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.366941    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.367218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.366995    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgq8\" (UniqueName: \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.367218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367049    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.367218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367073    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.367218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367126    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjkf\" (UniqueName: \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.367218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367160    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.367771 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367736    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.367893 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.367817    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.369593 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.369567    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.369681 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.369639    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.395855 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.395822    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:10.400080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.400051    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjkf\" (UniqueName: \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\") pod \"maas-api-revoke-sub-8b61c610-8cp2x\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.400800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.400774    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgq8\" (UniqueName: \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\") pod \"maas-api-revoke-sub-f1577dca-hhv9q\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.534183 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.534091    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:10.601598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.601559    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:10.920832 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:10.920801    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c459a9_32b5_4947_8b93_69685cd2c7ec.slice/crio-bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097 WatchSource:0}: Error finding container bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097: Status 404 returned error can't find the container with id bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097
Oct 30 08:04:10.921814 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.921789    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"]
Oct 30 08:04:10.950950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.950917    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x" event={"ID":"f5c459a9-32b5-4947-8b93-69685cd2c7ec","Type":"ContainerStarted","Data":"bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097"}
Oct 30 08:04:10.955586 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:10.955555    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"]
Oct 30 08:04:10.956598 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:10.956568    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebddd9f5_5154_4724_a1dc_db09ab270fcf.slice/crio-431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea WatchSource:0}: Error finding container 431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea: Status 404 returned error can't find the container with id 431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea
Oct 30 08:04:11.224832 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.224741    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:11.381317 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.378937    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"]
Oct 30 08:04:11.411320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.397344    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:11.411320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.407985    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe2ed-4s64n"]
Oct 30 08:04:11.690085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.690053    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1304a3ec-0e83-4205-ab4b-be0d16fc7194" path="/var/lib/kubelet/pods/1304a3ec-0e83-4205-ab4b-be0d16fc7194/volumes"
Oct 30 08:04:11.926164 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.926014    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:11.960785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.960740    6738 generic.go:356] "Generic (PLEG): container finished" podID="f5c459a9-32b5-4947-8b93-69685cd2c7ec" containerID="658ec37e7e253601d3b6ce59414acc867d21c8dcd85aeeb25058786bf67f9613" exitCode=0
Oct 30 08:04:11.960938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.960816    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x" event={"ID":"f5c459a9-32b5-4947-8b93-69685cd2c7ec","Type":"ContainerDied","Data":"658ec37e7e253601d3b6ce59414acc867d21c8dcd85aeeb25058786bf67f9613"}
Oct 30 08:04:11.962525 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.962502    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q" event={"ID":"ebddd9f5-5154-4724-a1dc-db09ab270fcf","Type":"ContainerStarted","Data":"935aaca4a2419d5638090f734807aff96264e4146657d80510eb2ef7f2374d4c"}
Oct 30 08:04:11.962661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:11.962544    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q" event={"ID":"ebddd9f5-5154-4724-a1dc-db09ab270fcf","Type":"ContainerStarted","Data":"431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea"}
Oct 30 08:04:12.058815 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.058706    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q" podStartSLOduration=2.058691774 podStartE2EDuration="2.058691774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:12.055517885 +0000 UTC m=+4012.488281778" watchObservedRunningTime="2026-10-30 08:04:12.058691774 +0000 UTC m=+4012.491455667"
Oct 30 08:04:12.565392 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.565334    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:12.732085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.732047    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:12.872016 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.871980    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:12.972347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.972287    6738 generic.go:356] "Generic (PLEG): container finished" podID="ebddd9f5-5154-4724-a1dc-db09ab270fcf" containerID="935aaca4a2419d5638090f734807aff96264e4146657d80510eb2ef7f2374d4c" exitCode=0
Oct 30 08:04:12.972347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:12.972324    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q" event={"ID":"ebddd9f5-5154-4724-a1dc-db09ab270fcf","Type":"ContainerDied","Data":"935aaca4a2419d5638090f734807aff96264e4146657d80510eb2ef7f2374d4c"}
Oct 30 08:04:13.131223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.131143    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:13.267286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.267246    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\") pod \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") "
Oct 30 08:04:13.267496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.267428    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\" (UniqueName: \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\") pod \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") "
Oct 30 08:04:13.267591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.267575    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\") pod \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\" (UID: \"f5c459a9-32b5-4947-8b93-69685cd2c7ec\") "
Oct 30 08:04:13.267949 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.267919    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca" pod "f5c459a9-32b5-4947-8b93-69685cd2c7ec" (UID: "f5c459a9-32b5-4947-8b93-69685cd2c7ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:13.268152 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.268134    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:13.269379 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.269357    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca" pod "f5c459a9-32b5-4947-8b93-69685cd2c7ec" (UID: "f5c459a9-32b5-4947-8b93-69685cd2c7ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:13.269529 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.269514    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf" pod "f5c459a9-32b5-4947-8b93-69685cd2c7ec" (UID: "f5c459a9-32b5-4947-8b93-69685cd2c7ec"). InnerVolumeSpecName "kube-api-access-smjkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:13.369871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.369834    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c459a9-32b5-4947-8b93-69685cd2c7ec-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:13.369871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.369865    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjkf\" (UniqueName: \"kubernetes.io/projected/f5c459a9-32b5-4947-8b93-69685cd2c7ec-kube-api-access-smjkf\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:13.452747 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.452663    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:13.980671 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.980640    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"
Oct 30 08:04:13.981122 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.980672    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x" event={"ID":"f5c459a9-32b5-4947-8b93-69685cd2c7ec","Type":"ContainerDied","Data":"bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097"}
Oct 30 08:04:13.981122 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:13.980723    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9005452abf628bed53438d0cb64d204ddeca7fb0bfb3095b948347f4f3f097"
Oct 30 08:04:14.137995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.137965    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:14.280690 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.280600    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\" (UniqueName: \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\") pod \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") "
Oct 30 08:04:14.280690 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.280679    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\") pod \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") "
Oct 30 08:04:14.280961 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.280711    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\") pod \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\" (UID: \"ebddd9f5-5154-4724-a1dc-db09ab270fcf\") "
Oct 30 08:04:14.281112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.281089    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca" pod "ebddd9f5-5154-4724-a1dc-db09ab270fcf" (UID: "ebddd9f5-5154-4724-a1dc-db09ab270fcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:14.281370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.281353    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:14.282851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.282831    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8" pod "ebddd9f5-5154-4724-a1dc-db09ab270fcf" (UID: "ebddd9f5-5154-4724-a1dc-db09ab270fcf"). InnerVolumeSpecName "kube-api-access-5lgq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:14.282932 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.282912    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca" pod "ebddd9f5-5154-4724-a1dc-db09ab270fcf" (UID: "ebddd9f5-5154-4724-a1dc-db09ab270fcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:14.382561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.382516    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgq8\" (UniqueName: \"kubernetes.io/projected/ebddd9f5-5154-4724-a1dc-db09ab270fcf-kube-api-access-5lgq8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:14.382561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.382554    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebddd9f5-5154-4724-a1dc-db09ab270fcf-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:14.988778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.988737    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"
Oct 30 08:04:14.989185 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.988778    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q" event={"ID":"ebddd9f5-5154-4724-a1dc-db09ab270fcf","Type":"ContainerDied","Data":"431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea"}
Oct 30 08:04:14.989185 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:14.988856    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="431fc03cd606c4e080375c343635a81d6823070155feb712b5fa8695b9f964ea"
Oct 30 08:04:15.333950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:15.333847    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:16.288361 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.288317    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"]
Oct 30 08:04:16.288933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.288918    6738 cpu_manager.go:414] "containerMap: removing container" podUID="f5c459a9-32b5-4947-8b93-69685cd2c7ec" containerName="revoke-keys"
Oct 30 08:04:16.288989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.288933    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c459a9-32b5-4947-8b93-69685cd2c7ec" containerName="revoke-keys"
Oct 30 08:04:16.288989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.288944    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ebddd9f5-5154-4724-a1dc-db09ab270fcf" containerName="revoke-keys"
Oct 30 08:04:16.288989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.288949    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebddd9f5-5154-4724-a1dc-db09ab270fcf" containerName="revoke-keys"
Oct 30 08:04:16.289111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.289070    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c459a9-32b5-4947-8b93-69685cd2c7ec" containerName="revoke-keys"
Oct 30 08:04:16.289111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.289077    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebddd9f5-5154-4724-a1dc-db09ab270fcf" containerName="revoke-keys"
Oct 30 08:04:16.292282 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.292265    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.310990 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.310958    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"]
Oct 30 08:04:16.319519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.319484    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:16.319519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.319483    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:16.421501 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.421458    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85m4\" (UniqueName: \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.422086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.422055    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.422383 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.422362    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.523617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.523579    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85m4\" (UniqueName: \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.523826 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.523694    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.523826 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.523747    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.524488 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.524468    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.526154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.526122    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.548188 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.548112    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85m4\" (UniqueName: \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\") pod \"maas-api-revoke-sub-5541922c-kmm5h\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.606614 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.606576    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:16.910366 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:16.910335    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"]
Oct 30 08:04:16.910509 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:16.910287    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28382c65_7d57_4e0e_a0e8_91a3f48a86c3.slice/crio-5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2 WatchSource:0}: Error finding container 5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2: Status 404 returned error can't find the container with id 5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2
Oct 30 08:04:17.004796 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.004747    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h" event={"ID":"28382c65-7d57-4e0e-a0e8-91a3f48a86c3","Type":"ContainerStarted","Data":"5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2"}
Oct 30 08:04:17.006926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.006897    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main" containerID="cri-o://19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66" gracePeriod=45
Oct 30 08:04:17.247624 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.247597    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 08:04:17.247806 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.247781    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.339938    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.340042    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.340171    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.340217    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\" (UniqueName: \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.340266    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.341139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.340393    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\") pod \"d4d1700b-5742-4b85-b755-44d4b104b5d1\" (UID: \"d4d1700b-5742-4b85-b755-44d4b104b5d1\") "
Oct 30 08:04:17.349926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.344901    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.349926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.345479    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.349926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.346641    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.352500 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.352461    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.352683 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.352509    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "kube-api-access-dlmxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:17.353955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.353927    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs" pod "d4d1700b-5742-4b85-b755-44d4b104b5d1" (UID: "d4d1700b-5742-4b85-b755-44d4b104b5d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442048    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442079    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442088    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d1700b-5742-4b85-b755-44d4b104b5d1-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442098    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmxp\" (UniqueName: \"kubernetes.io/projected/d4d1700b-5742-4b85-b755-44d4b104b5d1-kube-api-access-dlmxp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442107    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.442127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:17.442115    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d1700b-5742-4b85-b755-44d4b104b5d1-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:18.014753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.014671    6738 generic.go:356] "Generic (PLEG): container finished" podID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerID="19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66" exitCode=0
Oct 30 08:04:18.014753 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.014723    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"
Oct 30 08:04:18.015021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.014754    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" event={"ID":"d4d1700b-5742-4b85-b755-44d4b104b5d1","Type":"ContainerDied","Data":"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"}
Oct 30 08:04:18.015021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.014796    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f" event={"ID":"d4d1700b-5742-4b85-b755-44d4b104b5d1","Type":"ContainerDied","Data":"afa1bd1768ee6c16edc4f612fa718aa76c55d91df98c22c7319a1003cf69bf0a"}
Oct 30 08:04:18.015021 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.014815    6738 scope.go:122] "RemoveContainer" containerID="19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"
Oct 30 08:04:18.017896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.017866    6738 generic.go:356] "Generic (PLEG): container finished" podID="28382c65-7d57-4e0e-a0e8-91a3f48a86c3" containerID="f7e99be251e7d11ca68833b129e7d601789b350eeee86a10ee08e1a5781fa092" exitCode=0
Oct 30 08:04:18.018017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.017908    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h" event={"ID":"28382c65-7d57-4e0e-a0e8-91a3f48a86c3","Type":"ContainerDied","Data":"f7e99be251e7d11ca68833b129e7d601789b350eeee86a10ee08e1a5781fa092"}
Oct 30 08:04:18.026927 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.026902    6738 scope.go:122] "RemoveContainer" containerID="19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"
Oct 30 08:04:18.027305 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:18.027265    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66\": container with ID starting with 19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66 not found: ID does not exist" containerID="19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"
Oct 30 08:04:18.027423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.027319    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66"} err="failed to get container status \"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66\": rpc error: code = NotFound desc = could not find container \"19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66\": container with ID starting with 19c06bd715b046bf95c479f402cf2b6797cf5a58e07d33e8518560455fcc5e66 not found: ID does not exist"
Oct 30 08:04:18.074229 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.074188    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 08:04:18.086396 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:18.086355    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/distinct-2-w2-bb6be4-kserve-56b56cbbb9-jph8f"]
Oct 30 08:04:19.082265 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.082227    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:19.201619 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.201589    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:19.218874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.218842    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:19.367142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.367050    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\" (UniqueName: \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\") pod \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") "
Oct 30 08:04:19.367362 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.367168    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\") pod \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") "
Oct 30 08:04:19.367463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.367366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\") pod \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\" (UID: \"28382c65-7d57-4e0e-a0e8-91a3f48a86c3\") "
Oct 30 08:04:19.367603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.367578    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca" pod "28382c65-7d57-4e0e-a0e8-91a3f48a86c3" (UID: "28382c65-7d57-4e0e-a0e8-91a3f48a86c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:19.368077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.367987    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:19.371448 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.371414    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca" pod "28382c65-7d57-4e0e-a0e8-91a3f48a86c3" (UID: "28382c65-7d57-4e0e-a0e8-91a3f48a86c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:19.371448 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.371425    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4" pod "28382c65-7d57-4e0e-a0e8-91a3f48a86c3" (UID: "28382c65-7d57-4e0e-a0e8-91a3f48a86c3"). InnerVolumeSpecName "kube-api-access-g85m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:19.413049 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.413011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:19.469035 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.469002    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85m4\" (UniqueName: \"kubernetes.io/projected/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-kube-api-access-g85m4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:19.469035 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.469033    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28382c65-7d57-4e0e-a0e8-91a3f48a86c3-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:19.549079 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.549041    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:19.694551 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:19.694512    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" path="/var/lib/kubelet/pods/d4d1700b-5742-4b85-b755-44d4b104b5d1/volumes"
Oct 30 08:04:20.037766 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:20.037685    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h"
Oct 30 08:04:20.037766 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:20.037707    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5541922c-kmm5h" event={"ID":"28382c65-7d57-4e0e-a0e8-91a3f48a86c3","Type":"ContainerDied","Data":"5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2"}
Oct 30 08:04:20.037766 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:20.037760    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5caf013745c20ad536d79909fe7cf0b9caa33e7e2d8e9d2c8517ff8b2ccf2eb2"
Oct 30 08:04:20.080735 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:20.080696    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:20.208866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:20.208827    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:21.331320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:21.331271    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:23.288742 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:23.288709    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:25.219505 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:25.219467    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:26.648995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.648955    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"]
Oct 30 08:04:26.649548 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649533    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main"
Oct 30 08:04:26.649603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649549    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main"
Oct 30 08:04:26.649603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649570    6738 cpu_manager.go:414] "containerMap: removing container" podUID="28382c65-7d57-4e0e-a0e8-91a3f48a86c3" containerName="revoke-keys"
Oct 30 08:04:26.649603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649576    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28382c65-7d57-4e0e-a0e8-91a3f48a86c3" containerName="revoke-keys"
Oct 30 08:04:26.649709 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649671    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d1700b-5742-4b85-b755-44d4b104b5d1" containerName="main"
Oct 30 08:04:26.649709 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.649683    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="28382c65-7d57-4e0e-a0e8-91a3f48a86c3" containerName="revoke-keys"
Oct 30 08:04:26.652980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.652963    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.671933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.671904    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"]
Oct 30 08:04:26.680003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.679973    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:26.680171 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.680069    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:26.787541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.787498    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.787726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.787646    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqgw\" (UniqueName: \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.787831 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.787814    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.888946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.888905    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.889152 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.888963    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.889152 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.889036    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqgw\" (UniqueName: \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.889637 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.889613    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.891463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.891444    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.913424 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.913354    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqgw\" (UniqueName: \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\") pod \"maas-api-revoke-sub-2d81d4d0-ffpj5\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:26.967064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:26.967029    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:27.106392 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:27.106349    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:27.272027 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:27.271995    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"]
Oct 30 08:04:27.273196 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:27.273169    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f5ff51_8a58_4fe3_b7c6_2c956c849938.slice/crio-65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8 WatchSource:0}: Error finding container 65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8: Status 404 returned error can't find the container with id 65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8
Oct 30 08:04:28.097632 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:28.097595    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5" event={"ID":"70f5ff51-8a58-4fe3-b7c6-2c956c849938","Type":"ContainerStarted","Data":"6ce6258745e8b6c283dc8ae04bf45185cf768c0167fc245a44557e526beb09eb"}
Oct 30 08:04:28.098058 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:28.097646    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5" event={"ID":"70f5ff51-8a58-4fe3-b7c6-2c956c849938","Type":"ContainerStarted","Data":"65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8"}
Oct 30 08:04:28.139429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:28.139368    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5" podStartSLOduration=2.139355004 podStartE2EDuration="2.139355004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:28.138626204 +0000 UTC m=+4028.571390097" watchObservedRunningTime="2026-10-30 08:04:28.139355004 +0000 UTC m=+4028.572118896"
Oct 30 08:04:29.087949 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:29.087868    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:29.106840 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:29.106802    6738 generic.go:356] "Generic (PLEG): container finished" podID="70f5ff51-8a58-4fe3-b7c6-2c956c849938" containerID="6ce6258745e8b6c283dc8ae04bf45185cf768c0167fc245a44557e526beb09eb" exitCode=0
Oct 30 08:04:29.107264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:29.106887    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5" event={"ID":"70f5ff51-8a58-4fe3-b7c6-2c956c849938","Type":"ContainerDied","Data":"6ce6258745e8b6c283dc8ae04bf45185cf768c0167fc245a44557e526beb09eb"}
Oct 30 08:04:30.263244 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.263217    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:30.340315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.339813    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\") pod \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") "
Oct 30 08:04:30.340315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.339969    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\") pod \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") "
Oct 30 08:04:30.340315 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.340020    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\" (UniqueName: \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\") pod \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\" (UID: \"70f5ff51-8a58-4fe3-b7c6-2c956c849938\") "
Oct 30 08:04:30.349126 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.347605    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca" pod "70f5ff51-8a58-4fe3-b7c6-2c956c849938" (UID: "70f5ff51-8a58-4fe3-b7c6-2c956c849938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:30.349126 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.348718    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw" pod "70f5ff51-8a58-4fe3-b7c6-2c956c849938" (UID: "70f5ff51-8a58-4fe3-b7c6-2c956c849938"). InnerVolumeSpecName "kube-api-access-qcqgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:30.350541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.350510    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca" pod "70f5ff51-8a58-4fe3-b7c6-2c956c849938" (UID: "70f5ff51-8a58-4fe3-b7c6-2c956c849938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:30.441917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.441877    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:30.441917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.441913    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqgw\" (UniqueName: \"kubernetes.io/projected/70f5ff51-8a58-4fe3-b7c6-2c956c849938-kube-api-access-qcqgw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:30.442142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:30.441930    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f5ff51-8a58-4fe3-b7c6-2c956c849938-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:31.024168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.024136    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:31.122383 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.122344    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5" event={"ID":"70f5ff51-8a58-4fe3-b7c6-2c956c849938","Type":"ContainerDied","Data":"65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8"}
Oct 30 08:04:31.122631 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.122389    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d4d0-ffpj5"
Oct 30 08:04:31.122631 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.122403    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65cb031c067efff18216bc75668f1f9a3456c81a4b29b3da9ae7a7c0279f67d8"
Oct 30 08:04:31.531056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.531022    6738 scope.go:122] "RemoveContainer" containerID="19e4f994130dfc2a8350c3ae12674631f6ec1f9022719c54b7365456445f9eae"
Oct 30 08:04:31.541815 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:31.541788    6738 scope.go:122] "RemoveContainer" containerID="25f7dc47b4effb39b81756742dcee2a1b16e89f4652d55d212b225c9b0157eca"
Oct 30 08:04:32.948724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:32.948692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:33.326385 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.322958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:33.514750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.514709    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:33.643835 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.643794    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:33.680280 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.680236    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"]
Oct 30 08:04:33.691686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.691646    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb6be4-56ftp"]
Oct 30 08:04:33.782175 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:33.782141    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:34.214869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:34.214828    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:34.468615 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:34.468533    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:34.639245 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:34.639207    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:35.690480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:35.690439    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d842e4-9d28-4f35-a87c-3c7c283de05c" path="/var/lib/kubelet/pods/c4d842e4-9d28-4f35-a87c-3c7c283de05c/volumes"
Oct 30 08:04:35.763455 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:35.763416    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:36.167620 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:36.167583    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:37.521567 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:04:37.521530    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:04:37.730599 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.730560    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"]
Oct 30 08:04:37.731316 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.731283    6738 cpu_manager.go:414] "containerMap: removing container" podUID="70f5ff51-8a58-4fe3-b7c6-2c956c849938" containerName="revoke-keys"
Oct 30 08:04:37.731316 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.731317    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f5ff51-8a58-4fe3-b7c6-2c956c849938" containerName="revoke-keys"
Oct 30 08:04:37.731538 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.731436    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f5ff51-8a58-4fe3-b7c6-2c956c849938" containerName="revoke-keys"
Oct 30 08:04:37.736972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.736949    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.754083 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.754057    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:37.754246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.754126    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:37.769784 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.769748    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"]
Oct 30 08:04:37.789589 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.789483    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:37.844588 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.835138    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.844588 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.835325    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvfb\" (UniqueName: \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.844588 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.835481    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.936557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.936523    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvfb\" (UniqueName: \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.936723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.936610    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.936723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.936672    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.937307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.937266    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.939056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.939032    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:37.961168 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:37.961137    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvfb\" (UniqueName: \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\") pod \"maas-api-revoke-sub-359c2c46-g6kzn\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:38.053410 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:38.053326    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:38.354048 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:38.354016    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"]
Oct 30 08:04:38.355443 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:38.355411    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87624c52_50ff_4dea_a691_f41014ccbd37.slice/crio-25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03 WatchSource:0}: Error finding container 25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03: Status 404 returned error can't find the container with id 25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03
Oct 30 08:04:39.024102 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.024005    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:39.138062 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.138019    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:39.210536 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.210499    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn" event={"ID":"87624c52-50ff-4dea-a691-f41014ccbd37","Type":"ContainerStarted","Data":"52efcc345d8190b1c9b29d8d2a921223fda04a38bb0b22f9f3db3afc34ce20f2"}
Oct 30 08:04:39.210724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.210566    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn" event={"ID":"87624c52-50ff-4dea-a691-f41014ccbd37","Type":"ContainerStarted","Data":"25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03"}
Oct 30 08:04:39.251223 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.251165    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn" podStartSLOduration=2.251150604 podStartE2EDuration="2.251150604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:39.250938882 +0000 UTC m=+4039.683702777" watchObservedRunningTime="2026-10-30 08:04:39.251150604 +0000 UTC m=+4039.683914496"
Oct 30 08:04:39.761517 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:39.761346    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:40.220030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:40.219999    6738 generic.go:356] "Generic (PLEG): container finished" podID="87624c52-50ff-4dea-a691-f41014ccbd37" containerID="52efcc345d8190b1c9b29d8d2a921223fda04a38bb0b22f9f3db3afc34ce20f2" exitCode=0
Oct 30 08:04:40.220478 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:40.220087    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn" event={"ID":"87624c52-50ff-4dea-a691-f41014ccbd37","Type":"ContainerDied","Data":"52efcc345d8190b1c9b29d8d2a921223fda04a38bb0b22f9f3db3afc34ce20f2"}
Oct 30 08:04:41.012307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.012253    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:41.387320 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.387268    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:41.481170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.481136    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\") pod \"87624c52-50ff-4dea-a691-f41014ccbd37\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") "
Oct 30 08:04:41.481421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.481227    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\" (UniqueName: \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\") pod \"87624c52-50ff-4dea-a691-f41014ccbd37\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") "
Oct 30 08:04:41.481512 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.481422    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\") pod \"87624c52-50ff-4dea-a691-f41014ccbd37\" (UID: \"87624c52-50ff-4dea-a691-f41014ccbd37\") "
Oct 30 08:04:41.481746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.481724    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca" pod "87624c52-50ff-4dea-a691-f41014ccbd37" (UID: "87624c52-50ff-4dea-a691-f41014ccbd37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:41.481877 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.481857    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:41.483244 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.483220    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca" pod "87624c52-50ff-4dea-a691-f41014ccbd37" (UID: "87624c52-50ff-4dea-a691-f41014ccbd37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:41.483322 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.483284    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb" pod "87624c52-50ff-4dea-a691-f41014ccbd37" (UID: "87624c52-50ff-4dea-a691-f41014ccbd37"). InnerVolumeSpecName "kube-api-access-pvvfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:41.514126 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.514088    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:41.583663 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.583578    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvfb\" (UniqueName: \"kubernetes.io/projected/87624c52-50ff-4dea-a691-f41014ccbd37-kube-api-access-pvvfb\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:41.583663 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.583610    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87624c52-50ff-4dea-a691-f41014ccbd37-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:41.636443 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.636405    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:41.758117 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.758082    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:41.904791 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:41.904746    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:42.235570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:42.235487    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn"
Oct 30 08:04:42.235570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:42.235479    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359c2c46-g6kzn" event={"ID":"87624c52-50ff-4dea-a691-f41014ccbd37","Type":"ContainerDied","Data":"25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03"}
Oct 30 08:04:42.235814 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:42.235613    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d28e0424f89bcba1ca75a760a930dee92bb65e95418de9ac65dad8624dcd03"
Oct 30 08:04:43.719989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:43.719954    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:45.543505 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:45.543470    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:47.020746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:47.020713    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:47.415733 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:47.415700    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:48.322597 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.322566    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"]
Oct 30 08:04:48.323226 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.323209    6738 cpu_manager.go:414] "containerMap: removing container" podUID="87624c52-50ff-4dea-a691-f41014ccbd37" containerName="revoke-keys"
Oct 30 08:04:48.323275 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.323227    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87624c52-50ff-4dea-a691-f41014ccbd37" containerName="revoke-keys"
Oct 30 08:04:48.323364 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.323341    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="87624c52-50ff-4dea-a691-f41014ccbd37" containerName="revoke-keys"
Oct 30 08:04:48.327161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.327139    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.345145 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.345118    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"]
Oct 30 08:04:48.353688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.353659    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:48.353874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.353765    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:48.470469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.470432    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.470469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.470472    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.470785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.470619    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jsm\" (UniqueName: \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.572205 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.572170    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.572205 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.572210    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.572493 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.572289    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jsm\" (UniqueName: \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.573025 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.572969    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.574679 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.574657    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.596253 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.596221    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jsm\" (UniqueName: \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\") pod \"maas-api-revoke-sub-3f424632-4xtm2\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.641061 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.641025    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:48.948420 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:48.948391    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"]
Oct 30 08:04:48.950312 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:04:48.950272    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcdcee7_3dea_471a_bcda_106a5309a382.slice/crio-dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c WatchSource:0}: Error finding container dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c: Status 404 returned error can't find the container with id dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c
Oct 30 08:04:49.244817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:49.244719    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:49.319454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:49.319418    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2" event={"ID":"8bcdcee7-3dea-471a-bcda-106a5309a382","Type":"ContainerStarted","Data":"2748335e7f4b90aac9bb3a17ccf584b9bd944462bdd7214cd9a622d5e5161aad"}
Oct 30 08:04:49.319616 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:49.319475    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2" event={"ID":"8bcdcee7-3dea-471a-bcda-106a5309a382","Type":"ContainerStarted","Data":"dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c"}
Oct 30 08:04:49.361551 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:49.361480    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2" podStartSLOduration=1.361451907 podStartE2EDuration="1.361451907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:49.36083833 +0000 UTC m=+4049.793602225" watchObservedRunningTime="2026-10-30 08:04:49.361451907 +0000 UTC m=+4049.794215791"
Oct 30 08:04:50.328267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:50.328169    6738 generic.go:356] "Generic (PLEG): container finished" podID="8bcdcee7-3dea-471a-bcda-106a5309a382" containerID="2748335e7f4b90aac9bb3a17ccf584b9bd944462bdd7214cd9a622d5e5161aad" exitCode=0
Oct 30 08:04:50.328267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:50.328253    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2" event={"ID":"8bcdcee7-3dea-471a-bcda-106a5309a382","Type":"ContainerDied","Data":"2748335e7f4b90aac9bb3a17ccf584b9bd944462bdd7214cd9a622d5e5161aad"}
Oct 30 08:04:51.096147 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.096108    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:51.478835 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.478806    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:51.626644 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.626591    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\") pod \"8bcdcee7-3dea-471a-bcda-106a5309a382\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") "
Oct 30 08:04:51.626856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.626720    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\" (UniqueName: \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\") pod \"8bcdcee7-3dea-471a-bcda-106a5309a382\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") "
Oct 30 08:04:51.626856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.626797    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\") pod \"8bcdcee7-3dea-471a-bcda-106a5309a382\" (UID: \"8bcdcee7-3dea-471a-bcda-106a5309a382\") "
Oct 30 08:04:51.626982 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.626958    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca" pod "8bcdcee7-3dea-471a-bcda-106a5309a382" (UID: "8bcdcee7-3dea-471a-bcda-106a5309a382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:51.627307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.627275    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:51.628947 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.628918    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca" pod "8bcdcee7-3dea-471a-bcda-106a5309a382" (UID: "8bcdcee7-3dea-471a-bcda-106a5309a382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:51.629089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.629005    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm" pod "8bcdcee7-3dea-471a-bcda-106a5309a382" (UID: "8bcdcee7-3dea-471a-bcda-106a5309a382"). InnerVolumeSpecName "kube-api-access-f8jsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:51.728463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.728431    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcdcee7-3dea-471a-bcda-106a5309a382-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:51.728463 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:51.728457    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jsm\" (UniqueName: \"kubernetes.io/projected/8bcdcee7-3dea-471a-bcda-106a5309a382-kube-api-access-f8jsm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:04:52.343499 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:52.343466    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2"
Oct 30 08:04:52.343930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:52.343507    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f424632-4xtm2" event={"ID":"8bcdcee7-3dea-471a-bcda-106a5309a382","Type":"ContainerDied","Data":"dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c"}
Oct 30 08:04:52.343930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:52.343555    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf36957851a488ed833cc0fc03712ce3b795c55f338f6099bad8b1319e9d17c"
Oct 30 08:04:52.918603 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:52.918566    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:54.869974 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:54.869932    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:56.738839 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:56.738806    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:58.505677 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:58.505631    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:59.294417 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.294326    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:04:59.396968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.396932    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"]
Oct 30 08:04:59.398189 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.398154    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8bcdcee7-3dea-471a-bcda-106a5309a382" containerName="revoke-keys"
Oct 30 08:04:59.398412 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.398394    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcdcee7-3dea-471a-bcda-106a5309a382" containerName="revoke-keys"
Oct 30 08:04:59.398726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.398708    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcdcee7-3dea-471a-bcda-106a5309a382" containerName="revoke-keys"
Oct 30 08:04:59.407882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.407850    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.408518 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.408492    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"]
Oct 30 08:04:59.423667 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.423628    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:04:59.423795 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.423678    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:04:59.441419 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.441382    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.441624 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.441580    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.441708 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.441678    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg252\" (UniqueName: \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.543113 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.543080    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.543633 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.543191    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.543633 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.543246    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg252\" (UniqueName: \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.543849 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.543820    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.545749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.545682    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.566915 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.566884    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg252\" (UniqueName: \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\") pod \"maas-api-revoke-sub-2cc43303-6kvz2\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:04:59.726526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:04:59.726492    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:05:00.026827 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:00.026796    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"]
Oct 30 08:05:00.027335 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:05:00.027307    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bc6742_bffe_482d_a92a_1a934f39276e.slice/crio-9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637 WatchSource:0}: Error finding container 9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637: Status 404 returned error can't find the container with id 9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637
Oct 30 08:05:00.374736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:00.374702    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:00.455502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:00.455249    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2" event={"ID":"83bc6742-bffe-482d-a92a-1a934f39276e","Type":"ContainerStarted","Data":"20a6d1710204e8ea75dc0bbed010689370d5692ebaee3a2345d3c42218a666b2"}
Oct 30 08:05:00.455502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:00.455368    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2" event={"ID":"83bc6742-bffe-482d-a92a-1a934f39276e","Type":"ContainerStarted","Data":"9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637"}
Oct 30 08:05:00.497881 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:00.497824    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2" podStartSLOduration=1.497810505 podStartE2EDuration="1.497810505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:00.497188598 +0000 UTC m=+4060.929952490" watchObservedRunningTime="2026-10-30 08:05:00.497810505 +0000 UTC m=+4060.930574397"
Oct 30 08:05:01.463537 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:01.463501    6738 generic.go:356] "Generic (PLEG): container finished" podID="83bc6742-bffe-482d-a92a-1a934f39276e" containerID="20a6d1710204e8ea75dc0bbed010689370d5692ebaee3a2345d3c42218a666b2" exitCode=0
Oct 30 08:05:01.463965 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:01.463594    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2" event={"ID":"83bc6742-bffe-482d-a92a-1a934f39276e","Type":"ContainerDied","Data":"20a6d1710204e8ea75dc0bbed010689370d5692ebaee3a2345d3c42218a666b2"}
Oct 30 08:05:02.240305 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.240258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:02.628056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.628027    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:05:02.707633 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.707599    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\" (UniqueName: \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\") pod \"83bc6742-bffe-482d-a92a-1a934f39276e\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") "
Oct 30 08:05:02.707865 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.707659    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\") pod \"83bc6742-bffe-482d-a92a-1a934f39276e\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") "
Oct 30 08:05:02.707865 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.707754    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\") pod \"83bc6742-bffe-482d-a92a-1a934f39276e\" (UID: \"83bc6742-bffe-482d-a92a-1a934f39276e\") "
Oct 30 08:05:02.708076 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.708046    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca" pod "83bc6742-bffe-482d-a92a-1a934f39276e" (UID: "83bc6742-bffe-482d-a92a-1a934f39276e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:02.708362 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.708339    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:02.710080 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.710044    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252" pod "83bc6742-bffe-482d-a92a-1a934f39276e" (UID: "83bc6742-bffe-482d-a92a-1a934f39276e"). InnerVolumeSpecName "kube-api-access-bg252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:02.710187 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.710048    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca" pod "83bc6742-bffe-482d-a92a-1a934f39276e" (UID: "83bc6742-bffe-482d-a92a-1a934f39276e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:02.810051 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.809965    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg252\" (UniqueName: \"kubernetes.io/projected/83bc6742-bffe-482d-a92a-1a934f39276e-kube-api-access-bg252\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:02.810051 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:02.809999    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bc6742-bffe-482d-a92a-1a934f39276e-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:03.481010 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:03.480978    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2"
Oct 30 08:05:03.481170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:03.480980    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc43303-6kvz2" event={"ID":"83bc6742-bffe-482d-a92a-1a934f39276e","Type":"ContainerDied","Data":"9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637"}
Oct 30 08:05:03.481170 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:03.481102    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae28085c2636247a321b7b9e140c6ac53103e7073289b6703dbe01fc37bd637"
Oct 30 08:05:03.982648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:03.982609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:04.207135 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.207094    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"]
Oct 30 08:05:04.207763 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.207745    6738 cpu_manager.go:414] "containerMap: removing container" podUID="83bc6742-bffe-482d-a92a-1a934f39276e" containerName="revoke-keys"
Oct 30 08:05:04.207822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.207766    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bc6742-bffe-482d-a92a-1a934f39276e" containerName="revoke-keys"
Oct 30 08:05:04.207862 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.207856    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bc6742-bffe-482d-a92a-1a934f39276e" containerName="revoke-keys"
Oct 30 08:05:04.211500 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.211479    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.229825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.229800    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"]
Oct 30 08:05:04.233881 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.233805    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:04.237632 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.237604    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:05:04.237838 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.237818    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:05:04.358069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.358029    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5q6\" (UniqueName: \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.358264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.358097    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.358264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.358214    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.459898 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.459858    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5q6\" (UniqueName: \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.460108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.459922    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.460108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.459966    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.460660 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.460637    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.462459 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.462416    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.484264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.484192    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5q6\" (UniqueName: \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.526043 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.526013    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:04.835460 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:05:04.835421    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc076709_3e7e_40ac_b7ef_ac528f7b0b91.slice/crio-623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859 WatchSource:0}: Error finding container 623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859: Status 404 returned error can't find the container with id 623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859
Oct 30 08:05:04.836286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:04.836257    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"]
Oct 30 08:05:05.498447 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:05.498359    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s" event={"ID":"fc076709-3e7e-40ac-b7ef-ac528f7b0b91","Type":"ContainerStarted","Data":"60bd9b3a49351bfc8c554bfda384ea2211f859c580e741f571183f29cf6a0689"}
Oct 30 08:05:05.498447 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:05.498413    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s" event={"ID":"fc076709-3e7e-40ac-b7ef-ac528f7b0b91","Type":"ContainerStarted","Data":"623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859"}
Oct 30 08:05:05.579689 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:05.579634    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s" podStartSLOduration=1.579619327 podStartE2EDuration="1.579619327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:05.576112549 +0000 UTC m=+4066.008876441" watchObservedRunningTime="2026-10-30 08:05:05.579619327 +0000 UTC m=+4066.012383220"
Oct 30 08:05:06.365793 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:06.365709    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:06.507534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:06.507497    6738 generic.go:356] "Generic (PLEG): container finished" podID="fc076709-3e7e-40ac-b7ef-ac528f7b0b91" containerID="60bd9b3a49351bfc8c554bfda384ea2211f859c580e741f571183f29cf6a0689" exitCode=0
Oct 30 08:05:06.507953 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:06.507573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s" event={"ID":"fc076709-3e7e-40ac-b7ef-ac528f7b0b91","Type":"ContainerDied","Data":"60bd9b3a49351bfc8c554bfda384ea2211f859c580e741f571183f29cf6a0689"}
Oct 30 08:05:07.590983 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:05:07.590927    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:05:07.748549 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.748522    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:07.807043 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.807006    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\") pod \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") "
Oct 30 08:05:07.807241 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.807152    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\" (UniqueName: \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\") pod \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") "
Oct 30 08:05:07.807370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.807344    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\") pod \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\" (UID: \"fc076709-3e7e-40ac-b7ef-ac528f7b0b91\") "
Oct 30 08:05:07.807908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.807878    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca" pod "fc076709-3e7e-40ac-b7ef-ac528f7b0b91" (UID: "fc076709-3e7e-40ac-b7ef-ac528f7b0b91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:07.810494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.810457    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6" pod "fc076709-3e7e-40ac-b7ef-ac528f7b0b91" (UID: "fc076709-3e7e-40ac-b7ef-ac528f7b0b91"). InnerVolumeSpecName "kube-api-access-wb5q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:07.810748 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.810724    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca" pod "fc076709-3e7e-40ac-b7ef-ac528f7b0b91" (UID: "fc076709-3e7e-40ac-b7ef-ac528f7b0b91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:07.908812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.908774    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5q6\" (UniqueName: \"kubernetes.io/projected/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-kube-api-access-wb5q6\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:07.908812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.908807    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:07.908812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:07.908819    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076709-3e7e-40ac-b7ef-ac528f7b0b91-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:08.290140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:08.290100    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:08.565316 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:08.565217    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"
Oct 30 08:05:08.565316 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:08.565247    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s" event={"ID":"fc076709-3e7e-40ac-b7ef-ac528f7b0b91","Type":"ContainerDied","Data":"623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859"}
Oct 30 08:05:08.565528 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:08.565318    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623dc2e3cfe2422bc5a0ef66e171cf052ec889c77d807fd7e7f4a07d1785d859"
Oct 30 08:05:09.722886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.722840    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"]
Oct 30 08:05:09.723441 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.723426    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fc076709-3e7e-40ac-b7ef-ac528f7b0b91" containerName="revoke-keys"
Oct 30 08:05:09.723490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.723442    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc076709-3e7e-40ac-b7ef-ac528f7b0b91" containerName="revoke-keys"
Oct 30 08:05:09.723564 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.723518    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc076709-3e7e-40ac-b7ef-ac528f7b0b91" containerName="revoke-keys"
Oct 30 08:05:09.727143 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.727122    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.745281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.745250    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"]
Oct 30 08:05:09.754094 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.754062    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:05:09.754094 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.754065    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:05:09.830029 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.829994    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5649g\" (UniqueName: \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.830209 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.830051    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.830276 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.830227    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.931311 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.931262    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.931521 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.931379    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5649g\" (UniqueName: \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.931521 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.931419    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.931975 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.931942    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.933713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.933692    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:09.955667 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:09.955634    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5649g\" (UniqueName: \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\") pod \"maas-api-revoke-sub-8ed4f2cc-9lrhf\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:10.041803 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:10.041712    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:10.275917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:10.275883    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:10.348989 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:10.348961    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"]
Oct 30 08:05:10.349102 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:05:10.349000    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea3646c_555a_43fd_83e8_723ec893c8e4.slice/crio-64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa WatchSource:0}: Error finding container 64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa: Status 404 returned error can't find the container with id 64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa
Oct 30 08:05:10.580211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:10.580175    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf" event={"ID":"eea3646c-555a-43fd-83e8-723ec893c8e4","Type":"ContainerStarted","Data":"64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa"}
Oct 30 08:05:11.589405 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:11.589281    6738 generic.go:356] "Generic (PLEG): container finished" podID="eea3646c-555a-43fd-83e8-723ec893c8e4" containerID="64533d8edd173488b18ac4cd20d658bd88e5e36fe6cb71ac71537b3a868c936d" exitCode=0
Oct 30 08:05:11.589405 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:11.589324    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf" event={"ID":"eea3646c-555a-43fd-83e8-723ec893c8e4","Type":"ContainerDied","Data":"64533d8edd173488b18ac4cd20d658bd88e5e36fe6cb71ac71537b3a868c936d"}
Oct 30 08:05:12.261558 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.261515    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:12.747698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.747668    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:12.867501 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.867415    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\") pod \"eea3646c-555a-43fd-83e8-723ec893c8e4\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") "
Oct 30 08:05:12.867724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.867515    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\" (UniqueName: \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\") pod \"eea3646c-555a-43fd-83e8-723ec893c8e4\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") "
Oct 30 08:05:12.867724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.867557    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\") pod \"eea3646c-555a-43fd-83e8-723ec893c8e4\" (UID: \"eea3646c-555a-43fd-83e8-723ec893c8e4\") "
Oct 30 08:05:12.867955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.867932    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca" pod "eea3646c-555a-43fd-83e8-723ec893c8e4" (UID: "eea3646c-555a-43fd-83e8-723ec893c8e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:12.869679 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.869654    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca" pod "eea3646c-555a-43fd-83e8-723ec893c8e4" (UID: "eea3646c-555a-43fd-83e8-723ec893c8e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:12.869825 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.869805    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g" pod "eea3646c-555a-43fd-83e8-723ec893c8e4" (UID: "eea3646c-555a-43fd-83e8-723ec893c8e4"). InnerVolumeSpecName "kube-api-access-5649g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:12.969077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.969039    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:12.969077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.969072    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5649g\" (UniqueName: \"kubernetes.io/projected/eea3646c-555a-43fd-83e8-723ec893c8e4-kube-api-access-5649g\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:12.969077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:12.969083    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea3646c-555a-43fd-83e8-723ec893c8e4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:13.605327 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:13.605286    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf"
Oct 30 08:05:13.605504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:13.605322    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ed4f2cc-9lrhf" event={"ID":"eea3646c-555a-43fd-83e8-723ec893c8e4","Type":"ContainerDied","Data":"64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa"}
Oct 30 08:05:13.605504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:13.605377    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ca603a8613da59158ddfc94f972cdc54d2ddb73ba45ae064ff2c5cc474deaa"
Oct 30 08:05:14.112351 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:14.112287    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:14.699029 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:14.698990    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 08:05:14.813594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:14.813559    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:14.938449 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:14.938413    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:15.957784 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:15.957742    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:16.378197 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:16.378157    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:16.507954 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:16.507922    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:17.855186 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:17.855135    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:19.700173 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:19.700129    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing" containerID="cri-o://cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d" gracePeriod=25
Oct 30 08:05:19.844232 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:19.844192    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:20.713898 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:20.713847    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:05:20.780832 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:20.780791    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:21.510739 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:21.510695    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:22.129909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.129865    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:22.255421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.255373    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:22.273173 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.273136    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"]
Oct 30 08:05:22.284938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.284903    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d0c2f-mlr5s"]
Oct 30 08:05:22.375150 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.375110    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:22.773440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.773401    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:22.891672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:22.891637    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:23.689734 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:23.689699    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc076709-3e7e-40ac-b7ef-ac528f7b0b91" path="/var/lib/kubelet/pods/fc076709-3e7e-40ac-b7ef-ac528f7b0b91/volumes"
Oct 30 08:05:24.243965 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:24.243928    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:26.236547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:26.236510    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:28.148484 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:28.148450    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:28.790471 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:28.790429    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:29.323225 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:29.323185    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:29.450897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:29.450859    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:29.695289 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:29.695245    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:29.811972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:29.811937    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:30.103401 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.103283    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:30.283435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.283401    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"]
Oct 30 08:05:30.284108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.284087    6738 cpu_manager.go:414] "containerMap: removing container" podUID="eea3646c-555a-43fd-83e8-723ec893c8e4" containerName="revoke-keys"
Oct 30 08:05:30.284180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.284110    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3646c-555a-43fd-83e8-723ec893c8e4" containerName="revoke-keys"
Oct 30 08:05:30.284237 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.284228    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3646c-555a-43fd-83e8-723ec893c8e4" containerName="revoke-keys"
Oct 30 08:05:30.287474 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.287457    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.305451 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.305423    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"]
Oct 30 08:05:30.314263 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.314221    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:05:30.314474 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.314273    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:05:30.416410 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.416377    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.416410 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.416412    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.416895 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.416588    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld267\" (UniqueName: \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.517876 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.517836    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.517876 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.517872    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.518142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.517983    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld267\" (UniqueName: \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.518533 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.518510    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.520307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.520273    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.542614 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.542586    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld267\" (UniqueName: \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\") pod \"maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.602125 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.602088    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:30.905655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:30.905616    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"]
Oct 30 08:05:30.906336 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:05:30.906274    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445f3dab_1568_4352_9aaf_914f35b44a04.slice/crio-39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5 WatchSource:0}: Error finding container 39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5: Status 404 returned error can't find the container with id 39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5
Oct 30 08:05:31.619780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:31.619689    6738 scope.go:122] "RemoveContainer" containerID="ae52ab4243fb99def4495dc66b5c893414aa098a8a246bce8f14ba6cc677bef4"
Oct 30 08:05:31.746207 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:31.746168    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2" event={"ID":"445f3dab-1568-4352-9aaf-914f35b44a04","Type":"ContainerStarted","Data":"6b1f73f53fd852d79cb4c8f0aaca8148180996c4e7f7cc19fb3421cc6dd18d7a"}
Oct 30 08:05:31.746461 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:31.746227    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2" event={"ID":"445f3dab-1568-4352-9aaf-914f35b44a04","Type":"ContainerStarted","Data":"39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5"}
Oct 30 08:05:31.789034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:31.788976    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2" podStartSLOduration=1.788960754 podStartE2EDuration="1.788960754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:31.787447902 +0000 UTC m=+4092.220211794" watchObservedRunningTime="2026-10-30 08:05:31.788960754 +0000 UTC m=+4092.221724644"
Oct 30 08:05:32.218968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:32.218878    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:32.755469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:32.755434    6738 generic.go:356] "Generic (PLEG): container finished" podID="445f3dab-1568-4352-9aaf-914f35b44a04" containerID="6b1f73f53fd852d79cb4c8f0aaca8148180996c4e7f7cc19fb3421cc6dd18d7a" exitCode=0
Oct 30 08:05:32.755917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:32.755524    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2" event={"ID":"445f3dab-1568-4352-9aaf-914f35b44a04","Type":"ContainerDied","Data":"6b1f73f53fd852d79cb4c8f0aaca8148180996c4e7f7cc19fb3421cc6dd18d7a"}
Oct 30 08:05:33.914421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.914390    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:33.958904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.958866    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\" (UniqueName: \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\") pod \"445f3dab-1568-4352-9aaf-914f35b44a04\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") "
Oct 30 08:05:33.959194 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.959169    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\") pod \"445f3dab-1568-4352-9aaf-914f35b44a04\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") "
Oct 30 08:05:33.959333 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.959281    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\") pod \"445f3dab-1568-4352-9aaf-914f35b44a04\" (UID: \"445f3dab-1568-4352-9aaf-914f35b44a04\") "
Oct 30 08:05:33.959822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.959761    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca" pod "445f3dab-1568-4352-9aaf-914f35b44a04" (UID: "445f3dab-1568-4352-9aaf-914f35b44a04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:33.959953 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.959932    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:33.961732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.961703    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267" pod "445f3dab-1568-4352-9aaf-914f35b44a04" (UID: "445f3dab-1568-4352-9aaf-914f35b44a04"). InnerVolumeSpecName "kube-api-access-ld267". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:33.961863 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:33.961777    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca" pod "445f3dab-1568-4352-9aaf-914f35b44a04" (UID: "445f3dab-1568-4352-9aaf-914f35b44a04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:34.060742 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.060652    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld267\" (UniqueName: \"kubernetes.io/projected/445f3dab-1568-4352-9aaf-914f35b44a04-kube-api-access-ld267\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:34.060742 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.060682    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445f3dab-1568-4352-9aaf-914f35b44a04-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:34.129674 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.129636    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:34.771590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.771551    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2" event={"ID":"445f3dab-1568-4352-9aaf-914f35b44a04","Type":"ContainerDied","Data":"39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5"}
Oct 30 08:05:34.771590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.771588    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"
Oct 30 08:05:34.771826 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:34.771615    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39070d998c4efe79331756c543b911326bf584cd23a5c2a6f245ffecf3ae16c5"
Oct 30 08:05:35.737876 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:35.737829    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" podUID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" containerName="main" containerID="cri-o://e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5" gracePeriod=45
Oct 30 08:05:36.009995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.009969    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:05:36.084386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084349    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.084607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084406    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.084607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084498    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\" (UniqueName: \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.084607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084566    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.084607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084607    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.084888 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084662    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:36.084888 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084686    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\") pod \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\" (UID: \"5ff2db4b-8b43-4210-a92c-de75c4b7a689\") "
Oct 30 08:05:36.085089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.084897    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:36.085089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.085052    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:36.085373 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.085349    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.085492 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.085374    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.085492 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.085390    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.087022 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.086998    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "kube-api-access-hkhgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:36.087271 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.087255    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:36.087387 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.087264    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs" pod "5ff2db4b-8b43-4210-a92c-de75c4b7a689" (UID: "5ff2db4b-8b43-4210-a92c-de75c4b7a689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:36.186887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.186856    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2db4b-8b43-4210-a92c-de75c4b7a689-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.186887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.186884    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2db4b-8b43-4210-a92c-de75c4b7a689-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.186887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.186894    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhgw\" (UniqueName: \"kubernetes.io/projected/5ff2db4b-8b43-4210-a92c-de75c4b7a689-kube-api-access-hkhgw\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:36.243114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.243029    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:36.428994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.428953    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:36.787493 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.787457    6738 generic.go:356] "Generic (PLEG): container finished" podID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" containerID="e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5" exitCode=0
Oct 30 08:05:36.788006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.787502    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"
Oct 30 08:05:36.788006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.787545    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" event={"ID":"5ff2db4b-8b43-4210-a92c-de75c4b7a689","Type":"ContainerDied","Data":"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"}
Oct 30 08:05:36.788006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.787591    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g" event={"ID":"5ff2db4b-8b43-4210-a92c-de75c4b7a689","Type":"ContainerDied","Data":"6873353b778af0a307918d265dc847bd80bccc5182da69f349d961a87329c0fa"}
Oct 30 08:05:36.788006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.787614    6738 scope.go:122] "RemoveContainer" containerID="e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"
Oct 30 08:05:36.799180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.799158    6738 scope.go:122] "RemoveContainer" containerID="e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"
Oct 30 08:05:36.799457 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:05:36.799437    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5\": container with ID starting with e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5 not found: ID does not exist" containerID="e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"
Oct 30 08:05:36.799518 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.799466    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5"} err="failed to get container status \"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5\": rpc error: code = NotFound desc = could not find container \"e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5\": container with ID starting with e146ab8409d970ade97b10843c4fb14bc2a76ca381307428d1894ae578d9cde5 not found: ID does not exist"
Oct 30 08:05:36.844678 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.844638    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:05:36.856186 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:36.856141    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/rate-test-model-14271d-kserve-7ff4fc5597-7nb2g"]
Oct 30 08:05:37.519402 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:05:37.519368    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:05:37.689527 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:37.689492    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" path="/var/lib/kubelet/pods/5ff2db4b-8b43-4210-a92c-de75c4b7a689/volumes"
Oct 30 08:05:38.103512 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:38.103474    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:38.789889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:38.789850    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:39.957768 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:39.957689    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:39.963396 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:39.963370    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 08:05:40.028921 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.028884    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\" (UniqueName: \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\") pod \"7efdbb2a-df9f-433f-9a02-057101bbf985\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") "
Oct 30 08:05:40.029086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.028967    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\") pod \"7efdbb2a-df9f-433f-9a02-057101bbf985\" (UID: \"7efdbb2a-df9f-433f-9a02-057101bbf985\") "
Oct 30 08:05:40.029639 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.029615    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume" pod "7efdbb2a-df9f-433f-9a02-057101bbf985" (UID: "7efdbb2a-df9f-433f-9a02-057101bbf985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:40.031688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.031651    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v" pod "7efdbb2a-df9f-433f-9a02-057101bbf985" (UID: "7efdbb2a-df9f-433f-9a02-057101bbf985"). InnerVolumeSpecName "kube-api-access-zjq5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:40.130329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.130278    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq5v\" (UniqueName: \"kubernetes.io/projected/7efdbb2a-df9f-433f-9a02-057101bbf985-kube-api-access-zjq5v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:40.130329 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.130325    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efdbb2a-df9f-433f-9a02-057101bbf985-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:40.512889 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.512849    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 08:05:40.661590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.661554    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:40.786491 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.785803    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:40.821573 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.821541    6738 generic.go:356] "Generic (PLEG): container finished" podID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerID="cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d" exitCode=0
Oct 30 08:05:40.821800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.821585    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"
Oct 30 08:05:40.821800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.821598    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" event={"ID":"7efdbb2a-df9f-433f-9a02-057101bbf985","Type":"ContainerDied","Data":"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"}
Oct 30 08:05:40.821800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.821646    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh" event={"ID":"7efdbb2a-df9f-433f-9a02-057101bbf985","Type":"ContainerDied","Data":"6214ddca4a9fa36732ce1228d85c3e0fd514c39b50a781d8a407685243c489af"}
Oct 30 08:05:40.821800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.821674    6738 scope.go:122] "RemoveContainer" containerID="cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"
Oct 30 08:05:40.834617 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.834593    6738 scope.go:122] "RemoveContainer" containerID="cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"
Oct 30 08:05:40.834976 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:05:40.834941    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d\": container with ID starting with cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d not found: ID does not exist" containerID="cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"
Oct 30 08:05:40.835072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.834984    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d"} err="failed to get container status \"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d\": rpc error: code = NotFound desc = could not find container \"cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d\": container with ID starting with cf2919fbacf122d52fab626efc1ddafd7b7b9120402e4c65eefdcad025cd479d not found: ID does not exist"
Oct 30 08:05:40.876910 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.876873    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 08:05:40.881561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:40.881528    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d0c2f-54f5485547-qddzh"]
Oct 30 08:05:41.276285 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.276252    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 08:05:41.309908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.309874    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 08:05:41.427180 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.427145    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:41.548371 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.547635    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:41.691043 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.691005    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" path="/var/lib/kubelet/pods/7efdbb2a-df9f-433f-9a02-057101bbf985/volumes"
Oct 30 08:05:41.872105 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:41.872075    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:43.682394 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:43.682356    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:45.330646 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:45.330608    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:45.523475 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:45.523403    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing" containerID="cri-o://e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e" gracePeriod=25
Oct 30 08:05:45.924550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:45.924509    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:46.057275 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.057241    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:46.068688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.068654    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"]
Oct 30 08:05:46.080905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.080874    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e27e0c-mgxm2"]
Oct 30 08:05:46.179286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.179201    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:46.277086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.277040    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing" containerID="cri-o://ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936" gracePeriod=25
Oct 30 08:05:46.310157 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.310118    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing" containerID="cri-o://b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc" gracePeriod=25
Oct 30 08:05:46.595871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:46.595775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:47.071450 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:47.071411    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:47.690045 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:47.690010    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445f3dab-1568-4352-9aaf-914f35b44a04" path="/var/lib/kubelet/pods/445f3dab-1568-4352-9aaf-914f35b44a04/volumes"
Oct 30 08:05:48.045721 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:48.045620    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:49.679551 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:49.679507    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:49.680537 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:49.679666    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:49.884147 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:49.884105    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:51.834780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:51.834745    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:53.089468 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:53.089434    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:53.215595 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:53.215561    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:53.577807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:53.577774    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:53.869644 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:53.869557    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:55.980852 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:55.980813    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:56.143503 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:56.143469    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:57.071782 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.071743    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:57.626672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.626639    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 08:05:57.626914 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.626889    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" podUID="5938677c-2476-4e6f-a1f3-3192c897a5eb" containerName="maas-api" containerID="cri-o://42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf" gracePeriod=30
Oct 30 08:05:57.748897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.748854    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:57.926166 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.925787    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 08:05:57.968167 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968134    6738 generic.go:356] "Generic (PLEG): container finished" podID="5938677c-2476-4e6f-a1f3-3192c897a5eb" containerID="42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf" exitCode=0
Oct 30 08:05:57.968372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968181    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"
Oct 30 08:05:57.968372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968222    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" event={"ID":"5938677c-2476-4e6f-a1f3-3192c897a5eb","Type":"ContainerDied","Data":"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"}
Oct 30 08:05:57.968372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968277    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c" event={"ID":"5938677c-2476-4e6f-a1f3-3192c897a5eb","Type":"ContainerDied","Data":"794c852b200fa8f10f9ef0799332849b87b36636b993e2d0c8d35f0604acafaf"}
Oct 30 08:05:57.968372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968342    6738 scope.go:122] "RemoveContainer" containerID="42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"
Oct 30 08:05:57.968671 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968655    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\" (UniqueName: \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\") pod \"5938677c-2476-4e6f-a1f3-3192c897a5eb\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") "
Oct 30 08:05:57.968871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968855    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\") pod \"5938677c-2476-4e6f-a1f3-3192c897a5eb\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") "
Oct 30 08:05:57.968956 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968931    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\") pod \"5938677c-2476-4e6f-a1f3-3192c897a5eb\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") "
Oct 30 08:05:57.969006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.968956    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\") pod \"5938677c-2476-4e6f-a1f3-3192c897a5eb\" (UID: \"5938677c-2476-4e6f-a1f3-3192c897a5eb\") "
Oct 30 08:05:57.969477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.969452    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca" pod "5938677c-2476-4e6f-a1f3-3192c897a5eb" (UID: "5938677c-2476-4e6f-a1f3-3192c897a5eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:57.972703 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.972678    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5" pod "5938677c-2476-4e6f-a1f3-3192c897a5eb" (UID: "5938677c-2476-4e6f-a1f3-3192c897a5eb"). InnerVolumeSpecName "kube-api-access-tbqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:57.980684 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.980655    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls" pod "5938677c-2476-4e6f-a1f3-3192c897a5eb" (UID: "5938677c-2476-4e6f-a1f3-3192c897a5eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:57.980805 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:57.980717    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs" pod "5938677c-2476-4e6f-a1f3-3192c897a5eb" (UID: "5938677c-2476-4e6f-a1f3-3192c897a5eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:58.000514 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.000486    6738 scope.go:122] "RemoveContainer" containerID="42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"
Oct 30 08:05:58.000897 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:05:58.000873    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf\": container with ID starting with 42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf not found: ID does not exist" containerID="42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"
Oct 30 08:05:58.001017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.000905    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf"} err="failed to get container status \"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf\": rpc error: code = NotFound desc = could not find container \"42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf\": container with ID starting with 42cab7d848e06e9d90adf1fe3524a274f8774cac196126542906ed93455a89cf not found: ID does not exist"
Oct 30 08:05:58.069971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.069937    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:58.069971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.069969    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5938677c-2476-4e6f-a1f3-3192c897a5eb-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:58.069971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.069981    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqx5\" (UniqueName: \"kubernetes.io/projected/5938677c-2476-4e6f-a1f3-3192c897a5eb-kube-api-access-tbqx5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:58.070234 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.069989    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5938677c-2476-4e6f-a1f3-3192c897a5eb-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:05:58.317152 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.317080    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 08:05:58.329588 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.329558    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f5081-685888bbf6-fc59c"]
Oct 30 08:05:58.359496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.359462    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:58.481182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.481146    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:58.598733 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:58.598644    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:59.105425 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:59.105386    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:05:59.667165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:59.667129    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:59.667165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:59.667155    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:59.689985 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:05:59.689955    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5938677c-2476-4e6f-a1f3-3192c897a5eb" path="/var/lib/kubelet/pods/5938677c-2476-4e6f-a1f3-3192c897a5eb/volumes"
Oct 30 08:06:00.639944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:00.639906    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:02.644780 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:02.644727    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:04.431456 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:04.431418    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:04.622970 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:04.622934    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 08:06:04.737067 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:04.736962    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:04.852682 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:04.852643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:05.795711 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.795636    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 08:06:05.796148 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.796005    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:05.914416 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.914371    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\") pod \"e4e932aa-c64e-4390-87f7-b59305f4641b\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") "
Oct 30 08:06:05.914614 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.914525    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\" (UniqueName: \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\") pod \"e4e932aa-c64e-4390-87f7-b59305f4641b\" (UID: \"e4e932aa-c64e-4390-87f7-b59305f4641b\") "
Oct 30 08:06:05.914933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.914909    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume" pod "e4e932aa-c64e-4390-87f7-b59305f4641b" (UID: "e4e932aa-c64e-4390-87f7-b59305f4641b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:05.915154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.915137    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e932aa-c64e-4390-87f7-b59305f4641b-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:05.916735 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.916713    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg" pod "e4e932aa-c64e-4390-87f7-b59305f4641b" (UID: "e4e932aa-c64e-4390-87f7-b59305f4641b"). InnerVolumeSpecName "kube-api-access-ls9fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:05.929785 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:05.929746    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:06.016222 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.016181    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9fg\" (UniqueName: \"kubernetes.io/projected/e4e932aa-c64e-4390-87f7-b59305f4641b-kube-api-access-ls9fg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.030801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.030755    6738 generic.go:356] "Generic (PLEG): container finished" podID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerID="e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e" exitCode=0
Oct 30 08:06:06.030801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.030798    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"
Oct 30 08:06:06.031087 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.030825    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" event={"ID":"e4e932aa-c64e-4390-87f7-b59305f4641b","Type":"ContainerDied","Data":"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"}
Oct 30 08:06:06.031087 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.030867    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v" event={"ID":"e4e932aa-c64e-4390-87f7-b59305f4641b","Type":"ContainerDied","Data":"60ddf915cb3ab4a7d89b2b6aca2d0cd7f623fbb3db6b351929585c7881ddec5d"}
Oct 30 08:06:06.031087 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.030892    6738 scope.go:122] "RemoveContainer" containerID="e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"
Oct 30 08:06:06.042031 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.042011    6738 scope.go:122] "RemoveContainer" containerID="e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"
Oct 30 08:06:06.042361 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:06.042337    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e\": container with ID starting with e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e not found: ID does not exist" containerID="e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"
Oct 30 08:06:06.042448 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.042373    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e"} err="failed to get container status \"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e\": rpc error: code = NotFound desc = could not find container \"e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e\": container with ID starting with e9ad9fc61296a7c01a02fd10b27773fe5625a0354a61120c3017b4e7cb955f0e not found: ID does not exist"
Oct 30 08:06:06.090720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.090240    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 08:06:06.095472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.095433    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e27e0c-66684d54f8-95c4v"]
Oct 30 08:06:06.122809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.122759    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 08:06:06.123717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.123675    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy" containerID="cri-o://8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31" gracePeriod=30
Oct 30 08:06:06.165748 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.165704    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:06.365220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.365191    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:06.592651 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.592622    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 08:06:06.624422 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.624396    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 08:06:06.707986 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.707953    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"]
Oct 30 08:06:06.708687 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708659    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing"
Oct 30 08:06:06.708687 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708684    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708707    6738 cpu_manager.go:414] "containerMap: removing container" podUID="445f3dab-1568-4352-9aaf-914f35b44a04" containerName="revoke-keys"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708713    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445f3dab-1568-4352-9aaf-914f35b44a04" containerName="revoke-keys"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708730    6738 cpu_manager.go:414] "containerMap: removing container" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708735    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708757    6738 cpu_manager.go:414] "containerMap: removing container" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708775    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708784    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" containerName="main"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708788    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" containerName="main"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708799    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5938677c-2476-4e6f-a1f3-3192c897a5eb" containerName="maas-api"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708803    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5938677c-2476-4e6f-a1f3-3192c897a5eb" containerName="maas-api"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708809    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing"
Oct 30 08:06:06.708913 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708816    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708931    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" containerName="payload-pre-processing"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708952    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff2db4b-8b43-4210-a92c-de75c4b7a689" containerName="main"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708964    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5938677c-2476-4e6f-a1f3-3192c897a5eb" containerName="maas-api"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708973    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efdbb2a-df9f-433f-9a02-057101bbf985" containerName="payload-pre-processing"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708985    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" containerName="payload-processing"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.708992    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="445f3dab-1568-4352-9aaf-914f35b44a04" containerName="revoke-keys"
Oct 30 08:06:06.709477 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.709007    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerName="payload-processing"
Oct 30 08:06:06.712600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.712578    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.728110 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.728067    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\") pod \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") "
Oct 30 08:06:06.728369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.728346    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\" (UniqueName: \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\") pod \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\" (UID: \"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd\") "
Oct 30 08:06:06.729357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.729327    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume" pod "31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" (UID: "31a262e8-dbc6-4082-bf1b-3dc1655d8ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:06.729905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.729872    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\") pod \"4295b2e2-3177-4e60-ba21-eb456273e819\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") "
Oct 30 08:06:06.730019 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.729961    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\" (UniqueName: \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\") pod \"4295b2e2-3177-4e60-ba21-eb456273e819\" (UID: \"4295b2e2-3177-4e60-ba21-eb456273e819\") "
Oct 30 08:06:06.730514 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.730487    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.731082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.731058    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume" pod "4295b2e2-3177-4e60-ba21-eb456273e819" (UID: "4295b2e2-3177-4e60-ba21-eb456273e819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:06.732754 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.732727    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx" pod "31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" (UID: "31a262e8-dbc6-4082-bf1b-3dc1655d8ebd"). InnerVolumeSpecName "kube-api-access-z9ncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:06.732886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.732785    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c" pod "4295b2e2-3177-4e60-ba21-eb456273e819" (UID: "4295b2e2-3177-4e60-ba21-eb456273e819"). InnerVolumeSpecName "kube-api-access-nmc4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:06.734091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.734067    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"]
Oct 30 08:06:06.821042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.821006    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:06:06.821042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.821035    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:06:06.831760 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.831724    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75tv\" (UniqueName: \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.831922 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.831768    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.831922 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.831885    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.832023 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.832011    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ncx\" (UniqueName: \"kubernetes.io/projected/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd-kube-api-access-z9ncx\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.832069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.832025    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295b2e2-3177-4e60-ba21-eb456273e819-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.832069 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.832035    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmc4c\" (UniqueName: \"kubernetes.io/projected/4295b2e2-3177-4e60-ba21-eb456273e819-kube-api-access-nmc4c\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.933643 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.933558    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.933813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.933666    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75tv\" (UniqueName: \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.933813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.933691    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.934435 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.934409    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.935968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.935942    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:06.963508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:06.963473    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75tv\" (UniqueName: \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\") pod \"maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:07.029809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.029773    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:07.040506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.040476    6738 generic.go:356] "Generic (PLEG): container finished" podID="4295b2e2-3177-4e60-ba21-eb456273e819" containerID="b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc" exitCode=0
Oct 30 08:06:07.040691 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.040530    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"
Oct 30 08:06:07.040691 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.040561    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" event={"ID":"4295b2e2-3177-4e60-ba21-eb456273e819","Type":"ContainerDied","Data":"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"}
Oct 30 08:06:07.040691 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.040629    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx" event={"ID":"4295b2e2-3177-4e60-ba21-eb456273e819","Type":"ContainerDied","Data":"a34883babbeb8c92313972d5ed11617809d139b910b06fe61448e260ffcce986"}
Oct 30 08:06:07.040691 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.040653    6738 scope.go:122] "RemoveContainer" containerID="b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"
Oct 30 08:06:07.043498 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.043468    6738 generic.go:356] "Generic (PLEG): container finished" podID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" containerID="ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936" exitCode=0
Oct 30 08:06:07.043637 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.043532    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"
Oct 30 08:06:07.043637 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.043548    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" event={"ID":"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd","Type":"ContainerDied","Data":"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"}
Oct 30 08:06:07.043637 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.043592    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb" event={"ID":"31a262e8-dbc6-4082-bf1b-3dc1655d8ebd","Type":"ContainerDied","Data":"a5ad6bb671db8ecb64183757999f5700a8fefb0977ead2f8d4b00560d1e17bb5"}
Oct 30 08:06:07.064629 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.064602    6738 scope.go:122] "RemoveContainer" containerID="b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"
Oct 30 08:06:07.065041 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:07.065020    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc\": container with ID starting with b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc not found: ID does not exist" containerID="b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"
Oct 30 08:06:07.065131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.065052    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc"} err="failed to get container status \"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc\": rpc error: code = NotFound desc = could not find container \"b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc\": container with ID starting with b6e9a953313517a1af6443420aacf43d4581cad45fa9592b867fd429e4b8f4bc not found: ID does not exist"
Oct 30 08:06:07.065131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.065078    6738 scope.go:122] "RemoveContainer" containerID="ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"
Oct 30 08:06:07.076211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.076188    6738 scope.go:122] "RemoveContainer" containerID="ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"
Oct 30 08:06:07.076599 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:07.076573    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936\": container with ID starting with ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936 not found: ID does not exist" containerID="ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"
Oct 30 08:06:07.076717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.076603    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936"} err="failed to get container status \"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936\": rpc error: code = NotFound desc = could not find container \"ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936\": container with ID starting with ad5562306355314447bf94a66a713cd1adc972164db9ccb62cb1846011f5a936 not found: ID does not exist"
Oct 30 08:06:07.127853 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.127823    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 08:06:07.138938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.138899    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f5081-8dd478cf6-g65xb"]
Oct 30 08:06:07.179040 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.179003    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 08:06:07.186164 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.186091    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f5081-8554d84d7-ncwwx"]
Oct 30 08:06:07.354456 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.354431    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"]
Oct 30 08:06:07.356594 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:06:07.356561    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa56ea1b_6bc3_4e19_8184_002c49c03d32.slice/crio-86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7 WatchSource:0}: Error finding container 86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7: Status 404 returned error can't find the container with id 86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7
Oct 30 08:06:07.519106 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:07.519026    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:06:07.689730 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.689697    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a262e8-dbc6-4082-bf1b-3dc1655d8ebd" path="/var/lib/kubelet/pods/31a262e8-dbc6-4082-bf1b-3dc1655d8ebd/volumes"
Oct 30 08:06:07.690138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.690124    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4295b2e2-3177-4e60-ba21-eb456273e819" path="/var/lib/kubelet/pods/4295b2e2-3177-4e60-ba21-eb456273e819/volumes"
Oct 30 08:06:07.690515 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:07.690502    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e932aa-c64e-4390-87f7-b59305f4641b" path="/var/lib/kubelet/pods/e4e932aa-c64e-4390-87f7-b59305f4641b/volumes"
Oct 30 08:06:08.056630 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:08.056592    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr" event={"ID":"fa56ea1b-6bc3-4e19-8184-002c49c03d32","Type":"ContainerStarted","Data":"964f672e37f7a61877c7c89a5733cbaca8f2801a1d0ff5410ca7e968e3e55321"}
Oct 30 08:06:08.057220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:08.056655    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr" event={"ID":"fa56ea1b-6bc3-4e19-8184-002c49c03d32","Type":"ContainerStarted","Data":"86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7"}
Oct 30 08:06:08.099089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:08.099022    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr" podStartSLOduration=2.099003176 podStartE2EDuration="2.099003176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:08.098155522 +0000 UTC m=+4128.530919412" watchObservedRunningTime="2026-10-30 08:06:08.099003176 +0000 UTC m=+4128.531767071"
Oct 30 08:06:08.389993 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:08.389958    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:09.066007 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:09.065969    6738 generic.go:356] "Generic (PLEG): container finished" podID="fa56ea1b-6bc3-4e19-8184-002c49c03d32" containerID="964f672e37f7a61877c7c89a5733cbaca8f2801a1d0ff5410ca7e968e3e55321" exitCode=0
Oct 30 08:06:09.066533 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:09.066059    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr" event={"ID":"fa56ea1b-6bc3-4e19-8184-002c49c03d32","Type":"ContainerDied","Data":"964f672e37f7a61877c7c89a5733cbaca8f2801a1d0ff5410ca7e968e3e55321"}
Oct 30 08:06:09.624267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:09.624225    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing" containerID="cri-o://c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db" gracePeriod=25
Oct 30 08:06:10.222647 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.222622    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:10.374673 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.374646    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\") pod \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") "
Oct 30 08:06:10.374873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.374694    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\" (UniqueName: \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\") pod \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") "
Oct 30 08:06:10.374873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.374733    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\") pod \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\" (UID: \"fa56ea1b-6bc3-4e19-8184-002c49c03d32\") "
Oct 30 08:06:10.375104 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.375079    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca" pod "fa56ea1b-6bc3-4e19-8184-002c49c03d32" (UID: "fa56ea1b-6bc3-4e19-8184-002c49c03d32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:10.376994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.376963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca" pod "fa56ea1b-6bc3-4e19-8184-002c49c03d32" (UID: "fa56ea1b-6bc3-4e19-8184-002c49c03d32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:10.377119 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.377006    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv" pod "fa56ea1b-6bc3-4e19-8184-002c49c03d32" (UID: "fa56ea1b-6bc3-4e19-8184-002c49c03d32"). InnerVolumeSpecName "kube-api-access-q75tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:10.444928 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.444889    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:10.475896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.475866    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:10.475896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.475895    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75tv\" (UniqueName: \"kubernetes.io/projected/fa56ea1b-6bc3-4e19-8184-002c49c03d32-kube-api-access-q75tv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:10.476108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:10.475906    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa56ea1b-6bc3-4e19-8184-002c49c03d32-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.031099 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.031062    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"]
Oct 30 08:06:11.043327 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.043285    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da4ac29-hfxbt"]
Oct 30 08:06:11.082897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.082860    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr" event={"ID":"fa56ea1b-6bc3-4e19-8184-002c49c03d32","Type":"ContainerDied","Data":"86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7"}
Oct 30 08:06:11.083100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.082909    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f4d87540a9f74043cb69d7c06b9d5cf03d2b949233854a3f02ccb01dc2d2f7"
Oct 30 08:06:11.083100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.082878    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"
Oct 30 08:06:11.386324 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.386282    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 08:06:11.487357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487321    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487414    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487468    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487550 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487545    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487828 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487581    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487828 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487613    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:11.487828 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487623    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487828 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487722    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.487828 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487792    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\") pod \"7d78a9cd-852b-4cea-91d4-ce66d8027322\" (UID: \"7d78a9cd-852b-4cea-91d4-ce66d8027322\") "
Oct 30 08:06:11.488081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487842    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:11.488081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487934    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:11.488081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487947    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:11.488081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.487963    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:11.488342 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.488324    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.488429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.488344    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.488429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.488354    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.488429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.488378    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.488429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.488391    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d78a9cd-852b-4cea-91d4-ce66d8027322-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.490024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.489997    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:06:11.490133 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.490062    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:11.490201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.490172    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:11.490248 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.490232    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt" pod "7d78a9cd-852b-4cea-91d4-ce66d8027322" (UID: "7d78a9cd-852b-4cea-91d4-ce66d8027322"). InnerVolumeSpecName "kube-api-access-zv5dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:11.589809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.589722    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.589809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.589753    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5dt\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-kube-api-access-zv5dt\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.589809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.589765    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.589809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.589774    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d78a9cd-852b-4cea-91d4-ce66d8027322-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:11.688779 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:11.688734    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd196d7-e2ee-415d-bff1-daa621aca270" path="/var/lib/kubelet/pods/cbd196d7-e2ee-415d-bff1-daa621aca270/volumes"
Oct 30 08:06:12.020570 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.020531    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:12.091392 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.091350    6738 generic.go:356] "Generic (PLEG): container finished" podID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerID="8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31" exitCode=0
Oct 30 08:06:12.091573 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.091408    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"
Oct 30 08:06:12.091573 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.091438    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerDied","Data":"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"}
Oct 30 08:06:12.091573 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.091499    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm" event={"ID":"7d78a9cd-852b-4cea-91d4-ce66d8027322","Type":"ContainerDied","Data":"2a77d03bebbddef133867ff129db13bcf47374b667174ad1db9e5bdeaa7f8cb3"}
Oct 30 08:06:12.091573 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.091521    6738 scope.go:122] "RemoveContainer" containerID="8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"
Oct 30 08:06:12.106696 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.106674    6738 scope.go:122] "RemoveContainer" containerID="f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"
Oct 30 08:06:12.116621 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.116599    6738 scope.go:122] "RemoveContainer" containerID="8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"
Oct 30 08:06:12.116939 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:12.116916    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31\": container with ID starting with 8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31 not found: ID does not exist" containerID="8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"
Oct 30 08:06:12.116995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.116951    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31"} err="failed to get container status \"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31\": rpc error: code = NotFound desc = could not find container \"8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31\": container with ID starting with 8b429d133424ffbb4f27e9797392894e834d4faa3ed9f0d32c2753b15d19da31 not found: ID does not exist"
Oct 30 08:06:12.116995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.116969    6738 scope.go:122] "RemoveContainer" containerID="f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"
Oct 30 08:06:12.117257 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:12.117235    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e\": container with ID starting with f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e not found: ID does not exist" containerID="f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"
Oct 30 08:06:12.117342 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.117264    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e"} err="failed to get container status \"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e\": rpc error: code = NotFound desc = could not find container \"f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e\": container with ID starting with f9ed7de1e55d769d0dcb46e6090cf233de887ec2428aa3efb0f45db82b4f6d2e not found: ID does not exist"
Oct 30 08:06:12.159092 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.159059    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 08:06:12.161776 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.161756    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-9tzrm"]
Oct 30 08:06:12.672345 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.672314    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:12.791557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.791524    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:12.912991 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:12.912956    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:13.371863 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:13.371830    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:13.689115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:13.689033    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" path="/var/lib/kubelet/pods/7d78a9cd-852b-4cea-91d4-ce66d8027322/volumes"
Oct 30 08:06:14.033834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:14.033750    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"]
Oct 30 08:06:14.045891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:14.045860    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77452d0-fgdf6"]
Oct 30 08:06:14.930614 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:14.930570    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:15.689508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:15.689475    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73626942-0d27-44a8-9d5d-13d4ce96404f" path="/var/lib/kubelet/pods/73626942-0d27-44a8-9d5d-13d4ce96404f/volumes"
Oct 30 08:06:16.848524 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:16.848481    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:17.214746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:17.214708    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 08:06:17.400738 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:17.400702    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:17.525769 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:17.525677    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:17.836616 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:17.836526    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:18.749966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:18.749916    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:19.444432 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:19.444380    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:19.540838 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:19.540793    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"]
Oct 30 08:06:19.551732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:19.551627    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ace8fb-9wfvr"]
Oct 30 08:06:19.684706 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:19.684671    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:19.694628 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:19.694547    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa56ea1b-6bc3-4e19-8184-002c49c03d32" path="/var/lib/kubelet/pods/fa56ea1b-6bc3-4e19-8184-002c49c03d32/volumes"
Oct 30 08:06:21.102635 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:21.102580    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:22.020087 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:22.020049    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:22.102939 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:22.102882    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"]
Oct 30 08:06:22.111537 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:22.111491    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665f3a71-x42wm"]
Oct 30 08:06:22.215823 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:22.215782    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing" containerID="cri-o://d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f" gracePeriod=25
Oct 30 08:06:23.283772 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.283733    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:23.469746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.469708    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:23.631568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.631532    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 08:06:23.631788 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.631758    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy" containerID="cri-o://8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47" gracePeriod=30
Oct 30 08:06:23.690235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.690189    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a431fec-7a10-4470-861a-ccb6092e6fd4" path="/var/lib/kubelet/pods/8a431fec-7a10-4470-861a-ccb6092e6fd4/volumes"
Oct 30 08:06:23.737686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:23.737652    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:24.258564 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.258531    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"]
Oct 30 08:06:24.259037 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259024    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259039    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259062    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259068    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259086    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fa56ea1b-6bc3-4e19-8184-002c49c03d32" containerName="revoke-keys"
Oct 30 08:06:24.259091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259092    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa56ea1b-6bc3-4e19-8184-002c49c03d32" containerName="revoke-keys"
Oct 30 08:06:24.259274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259154    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259165    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d78a9cd-852b-4cea-91d4-ce66d8027322" containerName="istio-proxy"
Oct 30 08:06:24.259274 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.259174    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa56ea1b-6bc3-4e19-8184-002c49c03d32" containerName="revoke-keys"
Oct 30 08:06:24.263936 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.263917    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.295958 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.295921    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"]
Oct 30 08:06:24.314851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.314818    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:06:24.314851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.314835    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:06:24.332673 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.332638    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.332864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.332833    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.332915 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.332882    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2wn\" (UniqueName: \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.433569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.433532    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.433781 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.433586    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2wn\" (UniqueName: \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.433781 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.433616    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.434369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.434344    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.436038 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.436018    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.475504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.475468    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2wn\" (UniqueName: \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.578101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.578020    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:24.755813 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.755771    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:24.791139 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.791101    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:24.880675 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:24.880640    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"]
Oct 30 08:06:24.881132 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:06:24.881110    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cf9068_b356_4129_a75d_d1d1914638c4.slice/crio-9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482 WatchSource:0}: Error finding container 9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482: Status 404 returned error can't find the container with id 9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482
Oct 30 08:06:25.032718 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:25.032680    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"]
Oct 30 08:06:25.036333 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:25.036304    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984a0e4a-shvpb"]
Oct 30 08:06:25.186750 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:25.186714    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn" event={"ID":"67cf9068-b356-4129-a75d-d1d1914638c4","Type":"ContainerStarted","Data":"9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482"}
Oct 30 08:06:25.690976 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:25.690940    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3905c7-e765-4bbe-8422-80c740069251" path="/var/lib/kubelet/pods/bd3905c7-e765-4bbe-8422-80c740069251/volumes"
Oct 30 08:06:26.195817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:26.195777    6738 generic.go:356] "Generic (PLEG): container finished" podID="67cf9068-b356-4129-a75d-d1d1914638c4" containerID="c5411f05a49432cf91e14a048b21fdfaf0c799459f6d65c28811d7ae28906cfd" exitCode=0
Oct 30 08:06:26.196033 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:26.195868    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn" event={"ID":"67cf9068-b356-4129-a75d-d1d1914638c4","Type":"ContainerDied","Data":"c5411f05a49432cf91e14a048b21fdfaf0c799459f6d65c28811d7ae28906cfd"}
Oct 30 08:06:26.792181 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:26.792140    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:27.359692 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.359664    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:27.381005 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.380962    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\" (UniqueName: \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\") pod \"67cf9068-b356-4129-a75d-d1d1914638c4\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") "
Oct 30 08:06:27.381201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.381045    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\") pod \"67cf9068-b356-4129-a75d-d1d1914638c4\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") "
Oct 30 08:06:27.381201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.381111    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\") pod \"67cf9068-b356-4129-a75d-d1d1914638c4\" (UID: \"67cf9068-b356-4129-a75d-d1d1914638c4\") "
Oct 30 08:06:27.381561 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.381537    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca" pod "67cf9068-b356-4129-a75d-d1d1914638c4" (UID: "67cf9068-b356-4129-a75d-d1d1914638c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:27.383154 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.383132    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn" pod "67cf9068-b356-4129-a75d-d1d1914638c4" (UID: "67cf9068-b356-4129-a75d-d1d1914638c4"). InnerVolumeSpecName "kube-api-access-hg2wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:27.383233 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.383180    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca" pod "67cf9068-b356-4129-a75d-d1d1914638c4" (UID: "67cf9068-b356-4129-a75d-d1d1914638c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:27.483086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.482993    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:27.483086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.483032    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cf9068-b356-4129-a75d-d1d1914638c4-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:27.483086 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.483042    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2wn\" (UniqueName: \"kubernetes.io/projected/67cf9068-b356-4129-a75d-d1d1914638c4-kube-api-access-hg2wn\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:27.836592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.836504    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:27.865326 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:27.865258    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:28.032923 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.032885    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:28.215809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.215775    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"
Oct 30 08:06:28.215809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.215802    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn" event={"ID":"67cf9068-b356-4129-a75d-d1d1914638c4","Type":"ContainerDied","Data":"9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482"}
Oct 30 08:06:28.216119 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.215881    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cac5c5694e4360de7e2a949f7cab709f24140335029f39e5564ac95f1cdd482"
Oct 30 08:06:28.366646 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.366608    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:28.888897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.888868    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 08:06:28.892834 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.892803    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"]
Oct 30 08:06:28.893378 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893361    6738 cpu_manager.go:414] "containerMap: removing container" podUID="67cf9068-b356-4129-a75d-d1d1914638c4" containerName="revoke-keys"
Oct 30 08:06:28.893378 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893379    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67cf9068-b356-4129-a75d-d1d1914638c4" containerName="revoke-keys"
Oct 30 08:06:28.893508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893393    6738 cpu_manager.go:414] "containerMap: removing container" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:06:28.893508 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893401    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:06:28.893592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893514    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:06:28.893592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.893522    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="67cf9068-b356-4129-a75d-d1d1914638c4" containerName="revoke-keys"
Oct 30 08:06:28.896598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.896576    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:28.917347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.916274    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:06:28.917347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.916356    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:06:28.920402 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:28.920364    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"]
Oct 30 08:06:29.001541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001454    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001507    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001541 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001540    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001645    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001725    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001809 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001786    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001852    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001849    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.001994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001883    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001912    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\") pod \"183fa159-dc30-4e2d-953e-e83ed0986556\" (UID: \"183fa159-dc30-4e2d-953e-e83ed0986556\") "
Oct 30 08:06:29.001994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.001918    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.002350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002069    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.002350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002198    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.002350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002261    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.002350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002274    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:29.002350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002324    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bw8\" (UniqueName: \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002433    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002552    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002568    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002581    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002592    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.003003 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.002677    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/183fa159-dc30-4e2d-953e-e83ed0986556-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.004658 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.004471    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:29.004658 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.004602    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:06:29.005253 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.005031    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "kube-api-access-ckt2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:29.005253 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.005037    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy" pod "183fa159-dc30-4e2d-953e-e83ed0986556" (UID: "183fa159-dc30-4e2d-953e-e83ed0986556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.022800 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.022762    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:29.104063 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104022    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.104260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104162    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.104260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104190    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bw8\" (UniqueName: \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.104260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104233    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/183fa159-dc30-4e2d-953e-e83ed0986556-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.104260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104243    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.104260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104254    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/183fa159-dc30-4e2d-953e-e83ed0986556-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.104529 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104269    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckt2m\" (UniqueName: \"kubernetes.io/projected/183fa159-dc30-4e2d-953e-e83ed0986556-kube-api-access-ckt2m\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.104899 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.104874    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.106572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.106556    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.128498 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.128465    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bw8\" (UniqueName: \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.211102 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.211020    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:29.225100 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.225054    6738 generic.go:356] "Generic (PLEG): container finished" podID="183fa159-dc30-4e2d-953e-e83ed0986556" containerID="8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47" exitCode=0
Oct 30 08:06:29.225201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.225103    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"
Oct 30 08:06:29.225201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.225134    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerDied","Data":"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"}
Oct 30 08:06:29.225201 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.225193    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9" event={"ID":"183fa159-dc30-4e2d-953e-e83ed0986556","Type":"ContainerDied","Data":"64e48dc94bdfb1a2ebb4d7746a8c7ac87f4b5c9758b8cf1a0f482402ad0149f9"}
Oct 30 08:06:29.225384 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.225217    6738 scope.go:122] "RemoveContainer" containerID="8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"
Oct 30 08:06:29.236818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.236798    6738 scope.go:122] "RemoveContainer" containerID="0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"
Oct 30 08:06:29.247361 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.247335    6738 scope.go:122] "RemoveContainer" containerID="8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"
Oct 30 08:06:29.247731 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:29.247706    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47\": container with ID starting with 8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47 not found: ID does not exist" containerID="8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"
Oct 30 08:06:29.247801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.247744    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47"} err="failed to get container status \"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47\": rpc error: code = NotFound desc = could not find container \"8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47\": container with ID starting with 8b774ac812da0cdb2b4b1a7272b851a35f5367051be0dd62a23f369a8f341b47 not found: ID does not exist"
Oct 30 08:06:29.247801 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.247762    6738 scope.go:122] "RemoveContainer" containerID="0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"
Oct 30 08:06:29.248056 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:29.248040    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69\": container with ID starting with 0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69 not found: ID does not exist" containerID="0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"
Oct 30 08:06:29.248102 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.248062    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69"} err="failed to get container status \"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69\": rpc error: code = NotFound desc = could not find container \"0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69\": container with ID starting with 0b78af8fe5bc4ec59d458af3fb83629aaf0a9d3664a1436551c6da1dd20b0c69 not found: ID does not exist"
Oct 30 08:06:29.285449 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.285416    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 08:06:29.288722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.288690    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-g5zw9"]
Oct 30 08:06:29.520575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.520543    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"]
Oct 30 08:06:29.521336 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:06:29.521307    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6b2f6e_493c_4df8_ad83_2dfd848ab723.slice/crio-15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc WatchSource:0}: Error finding container 15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc: Status 404 returned error can't find the container with id 15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc
Oct 30 08:06:29.690856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.690819    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" path="/var/lib/kubelet/pods/183fa159-dc30-4e2d-953e-e83ed0986556/volumes"
Oct 30 08:06:29.880089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.880058    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 08:06:29.913495 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.913453    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\" (UniqueName: \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\") pod \"1e276cef-464c-4b86-b80c-c26e265f57fb\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") "
Oct 30 08:06:29.914010 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.913599    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\") pod \"1e276cef-464c-4b86-b80c-c26e265f57fb\" (UID: \"1e276cef-464c-4b86-b80c-c26e265f57fb\") "
Oct 30 08:06:29.914087 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.914046    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume" pod "1e276cef-464c-4b86-b80c-c26e265f57fb" (UID: "1e276cef-464c-4b86-b80c-c26e265f57fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:29.915820 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:29.915795    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl" pod "1e276cef-464c-4b86-b80c-c26e265f57fb" (UID: "1e276cef-464c-4b86-b80c-c26e265f57fb"). InnerVolumeSpecName "kube-api-access-s4mgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:30.015282 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.015243    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mgl\" (UniqueName: \"kubernetes.io/projected/1e276cef-464c-4b86-b80c-c26e265f57fb-kube-api-access-s4mgl\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:30.015282 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.015274    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e276cef-464c-4b86-b80c-c26e265f57fb-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:30.235466 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.235430    6738 generic.go:356] "Generic (PLEG): container finished" podID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerID="c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db" exitCode=0
Oct 30 08:06:30.235698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.235477    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"
Oct 30 08:06:30.235698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.235518    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" event={"ID":"1e276cef-464c-4b86-b80c-c26e265f57fb","Type":"ContainerDied","Data":"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"}
Oct 30 08:06:30.235698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.235575    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc" event={"ID":"1e276cef-464c-4b86-b80c-c26e265f57fb","Type":"ContainerDied","Data":"b4364c27bb96ae88bfc760f0963e477200deb38ad25eea6d08b56b762ce00768"}
Oct 30 08:06:30.235698 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.235607    6738 scope.go:122] "RemoveContainer" containerID="c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"
Oct 30 08:06:30.237534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.237506    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s" event={"ID":"6a6b2f6e-493c-4df8-ad83-2dfd848ab723","Type":"ContainerStarted","Data":"53195bbb06ddd84990851785dbd8b5a41af90b6c62390d09b4f897e56ceb7eca"}
Oct 30 08:06:30.237649 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.237558    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s" event={"ID":"6a6b2f6e-493c-4df8-ad83-2dfd848ab723","Type":"ContainerStarted","Data":"15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc"}
Oct 30 08:06:30.246334 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.246286    6738 scope.go:122] "RemoveContainer" containerID="c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"
Oct 30 08:06:30.246636 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:30.246618    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db\": container with ID starting with c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db not found: ID does not exist" containerID="c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"
Oct 30 08:06:30.246712 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.246644    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db"} err="failed to get container status \"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db\": rpc error: code = NotFound desc = could not find container \"c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db\": container with ID starting with c8bc2f9fa2682a78038557feb7a0b6a3bfd8ac7386ab0e9cc2e643e327a7e4db not found: ID does not exist"
Oct 30 08:06:30.292157 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.292117    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 08:06:30.295539 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.295501    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bd8589-5fd7f86c9f-dnnwc"]
Oct 30 08:06:30.348077 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:30.348010    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s" podStartSLOduration=2.347993941 podStartE2EDuration="2.347993941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:30.345873385 +0000 UTC m=+4150.778637278" watchObservedRunningTime="2026-10-30 08:06:30.347993941 +0000 UTC m=+4150.780757835"
Oct 30 08:06:31.247442 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:31.247399    6738 generic.go:356] "Generic (PLEG): container finished" podID="6a6b2f6e-493c-4df8-ad83-2dfd848ab723" containerID="53195bbb06ddd84990851785dbd8b5a41af90b6c62390d09b4f897e56ceb7eca" exitCode=0
Oct 30 08:06:31.247851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:31.247482    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s" event={"ID":"6a6b2f6e-493c-4df8-ad83-2dfd848ab723","Type":"ContainerDied","Data":"53195bbb06ddd84990851785dbd8b5a41af90b6c62390d09b4f897e56ceb7eca"}
Oct 30 08:06:31.277932 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:31.277888    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:31.653696 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:31.653667    6738 scope.go:122] "RemoveContainer" containerID="3606e0b0fc4d4c7b50a269c1e85713178857d38b0146b2fae07522dfd3c0ca8e"
Oct 30 08:06:31.688547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:31.688522    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" path="/var/lib/kubelet/pods/1e276cef-464c-4b86-b80c-c26e265f57fb/volumes"
Oct 30 08:06:32.075900 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.075809    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"]
Oct 30 08:06:32.088368 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.088331    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c01c38-r47hc"]
Oct 30 08:06:32.401229 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.401201    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:32.447912 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.447870    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\") pod \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") "
Oct 30 08:06:32.448078 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.447987    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\") pod \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") "
Oct 30 08:06:32.448078 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.448049    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\" (UniqueName: \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\") pod \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\" (UID: \"6a6b2f6e-493c-4df8-ad83-2dfd848ab723\") "
Oct 30 08:06:32.448460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.448434    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca" pod "6a6b2f6e-493c-4df8-ad83-2dfd848ab723" (UID: "6a6b2f6e-493c-4df8-ad83-2dfd848ab723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:32.450072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.450048    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca" pod "6a6b2f6e-493c-4df8-ad83-2dfd848ab723" (UID: "6a6b2f6e-493c-4df8-ad83-2dfd848ab723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:32.450159 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.450138    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8" pod "6a6b2f6e-493c-4df8-ad83-2dfd848ab723" (UID: "6a6b2f6e-493c-4df8-ad83-2dfd848ab723"). InnerVolumeSpecName "kube-api-access-x9bw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:32.549955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.549916    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:32.549955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.549949    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:32.549955 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:32.549960    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bw8\" (UniqueName: \"kubernetes.io/projected/6a6b2f6e-493c-4df8-ad83-2dfd848ab723-kube-api-access-x9bw8\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:33.264822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:33.264787    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"
Oct 30 08:06:33.264822 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:33.264809    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s" event={"ID":"6a6b2f6e-493c-4df8-ad83-2dfd848ab723","Type":"ContainerDied","Data":"15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc"}
Oct 30 08:06:33.265066 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:33.264859    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15ec1c007a342a7cb1084285948b3f35c7bc7381344d3b32305d489203b646bc"
Oct 30 08:06:33.347415 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:33.347383    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:33.689590 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:33.689552    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0" path="/var/lib/kubelet/pods/48ddda41-f65b-4b9a-a9b7-4bd2d3f2e1d0/volumes"
Oct 30 08:06:34.755090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:34.755050    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:34.859526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:34.859490    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 08:06:34.996412 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:34.996372    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:35.134896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:35.134862    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:35.454818 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:35.454737    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:36.457950 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:36.457908    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:37.517985 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:37.517947    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:06:37.533547 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:37.533510    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:39.422382 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.422344    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 08:06:39.433733 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.433692    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:39.452904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.452864    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 08:06:39.577623 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.577589    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:39.691964 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.691869    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:39.860408 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:39.860359    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing" containerID="cri-o://8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef" gracePeriod=25
Oct 30 08:06:40.513421 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:40.513384    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:41.332933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:41.332900    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:42.480942 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.480877    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 08:06:42.566719 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.566680    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\" (UniqueName: \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\") pod \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") "
Oct 30 08:06:42.566904 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.566788    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\") pod \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\" (UID: \"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1\") "
Oct 30 08:06:42.567257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.567227    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume" pod "70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" (UID: "70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:42.567499 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.567480    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:42.568871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.568849    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9" pod "70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" (UID: "70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1"). InnerVolumeSpecName "kube-api-access-fbjd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:42.668524 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.668490    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjd9\" (UniqueName: \"kubernetes.io/projected/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1-kube-api-access-fbjd9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:42.890967 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:42.890927    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:43.029688 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.029648    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"]
Oct 30 08:06:43.034846 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.034813    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfc57b52-v5gpz"]
Oct 30 08:06:43.337108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.337018    6738 generic.go:356] "Generic (PLEG): container finished" podID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerID="d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f" exitCode=0
Oct 30 08:06:43.337108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.337083    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"
Oct 30 08:06:43.337108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.337100    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" event={"ID":"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1","Type":"ContainerDied","Data":"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"}
Oct 30 08:06:43.337371 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.337175    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn" event={"ID":"70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1","Type":"ContainerDied","Data":"64eaae2847c60f6a88f93916962feeb724400cc83afb5cbfe84bbc5742589814"}
Oct 30 08:06:43.337371 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.337200    6738 scope.go:122] "RemoveContainer" containerID="d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"
Oct 30 08:06:43.348367 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.348344    6738 scope.go:122] "RemoveContainer" containerID="d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"
Oct 30 08:06:43.348715 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:43.348690    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f\": container with ID starting with d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f not found: ID does not exist" containerID="d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"
Oct 30 08:06:43.348806 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.348724    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f"} err="failed to get container status \"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f\": rpc error: code = NotFound desc = could not find container \"d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f\": container with ID starting with d5c4151c3c94b8c6d0fcba1cfd489402c4cf2f8d1962eec5399df7d850dd8c8f not found: ID does not exist"
Oct 30 08:06:43.391191 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.391155    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 08:06:43.403505 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.403470    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ace8fb-567df497b8-tsqsn"]
Oct 30 08:06:43.490058 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.490026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:43.606754 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.606662    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:43.637909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.637875    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"]
Oct 30 08:06:43.644460 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.644424    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b5998f-d8fcn"]
Oct 30 08:06:43.704442 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.704411    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67cf9068-b356-4129-a75d-d1d1914638c4" path="/var/lib/kubelet/pods/67cf9068-b356-4129-a75d-d1d1914638c4/volumes"
Oct 30 08:06:43.704851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.704836    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" path="/var/lib/kubelet/pods/70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1/volumes"
Oct 30 08:06:43.705253 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.705239    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c31806-3f63-42d6-9898-02c8a5e2e14e" path="/var/lib/kubelet/pods/82c31806-3f63-42d6-9898-02c8a5e2e14e/volumes"
Oct 30 08:06:43.717526 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:43.717490    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:44.139746 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:44.139713    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:44.423439 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:44.423347    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing" containerID="cri-o://abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af" gracePeriod=25
Oct 30 08:06:44.454096 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:44.454020    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing" containerID="cri-o://f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4" gracePeriod=25
Oct 30 08:06:45.750635 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:45.750593    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:45.790935 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:45.790893    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:46.809730 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:46.809685    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:47.437545 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:47.437508    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:48.039686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:48.039648    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:48.161523 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:48.161482    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:48.183815 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:48.183778    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"]
Oct 30 08:06:48.196098 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:48.196066    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14271d-sqs7s"]
Oct 30 08:06:48.698370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:48.698336    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:49.689494 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:49.689454    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6b2f6e-493c-4df8-ad83-2dfd848ab723" path="/var/lib/kubelet/pods/6a6b2f6e-493c-4df8-ad83-2dfd848ab723/volumes"
Oct 30 08:06:50.161556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:50.161516    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:50.514218 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:50.514113    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:50.749246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:50.749203    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:50.873260 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:50.873227    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:51.151000 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:51.150903    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 08:06:51.151206 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:51.151167    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy" containerID="cri-o://dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70" gracePeriod=30
Oct 30 08:06:51.166181 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:51.166132    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:51.617101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:51.617011    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:52.375600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:52.375569    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:54.349682 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:54.349639    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:55.596136 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:55.596096    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:55.729805 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:55.729766    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:55.790704 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:55.790663    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:56.089749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.089707    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:56.242067 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.242024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:56.394108 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.394058    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:56.413615 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.413586    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 08:06:56.414366 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.414334    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" containerID="cri-o://3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b" gracePeriod=30
Oct 30 08:06:56.432303 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.432256    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 08:06:56.436219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.436177    6738 generic.go:356] "Generic (PLEG): container finished" podID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerID="dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70" exitCode=0
Oct 30 08:06:56.436429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.436242    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"
Oct 30 08:06:56.436429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.436249    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerDied","Data":"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"}
Oct 30 08:06:56.436600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.436487    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm" event={"ID":"11e1dc2d-08cd-4754-a629-4edc43dc612c","Type":"ContainerDied","Data":"2772bbf0936dded44f769ec54bea1f7e9d3fac9839a5aee0e6dabae0f79bdc1b"}
Oct 30 08:06:56.436600 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.436526    6738 scope.go:122] "RemoveContainer" containerID="dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"
Oct 30 08:06:56.443331 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443270    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443349    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443420    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443515    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443535    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443569    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443598    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443625    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.443648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443647    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\") pod \"11e1dc2d-08cd-4754-a629-4edc43dc612c\" (UID: \"11e1dc2d-08cd-4754-a629-4edc43dc612c\") "
Oct 30 08:06:56.444024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443828    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.444024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.443915    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.444155 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444024    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.444155 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444126    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.444281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444177    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.444281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444198    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.444281 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444214    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.444531 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.444361    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:56.446005 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.445973    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:56.446569 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.446542    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.446710 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.446622    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:06:56.448199 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.448169    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj" pod "11e1dc2d-08cd-4754-a629-4edc43dc612c" (UID: "11e1dc2d-08cd-4754-a629-4edc43dc612c"). InnerVolumeSpecName "kube-api-access-prdhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:56.450937 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.450877    6738 scope.go:122] "RemoveContainer" containerID="d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"
Oct 30 08:06:56.472195 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.472162    6738 scope.go:122] "RemoveContainer" containerID="dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"
Oct 30 08:06:56.472631 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:56.472608    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70\": container with ID starting with dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70 not found: ID does not exist" containerID="dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"
Oct 30 08:06:56.472732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.472640    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70"} err="failed to get container status \"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70\": rpc error: code = NotFound desc = could not find container \"dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70\": container with ID starting with dfbcba1f3678edaafa9d187b529c3ca9f0d1a6c4be475feaf9231fd7ddd22e70 not found: ID does not exist"
Oct 30 08:06:56.472732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.472663    6738 scope.go:122] "RemoveContainer" containerID="d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"
Oct 30 08:06:56.473038 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:06:56.473016    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c\": container with ID starting with d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c not found: ID does not exist" containerID="d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"
Oct 30 08:06:56.473091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.473045    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c"} err="failed to get container status \"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c\": rpc error: code = NotFound desc = could not find container \"d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c\": container with ID starting with d4ae6f76d336c59e47503cb44ed6086eaa683e9d0edb979b4a73389711301c4c not found: ID does not exist"
Oct 30 08:06:56.535634 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.535049    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:56.545686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545641    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdhj\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-kube-api-access-prdhj\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.545686 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545694    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.546001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545719    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.546001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545735    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1dc2d-08cd-4754-a629-4edc43dc612c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.546001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545757    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.546001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.545775    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1dc2d-08cd-4754-a629-4edc43dc612c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.566221 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.566185    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 08:06:56.706558 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.706442    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:06:56.784490 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.784454    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 08:06:56.789074 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.789040    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-hcpdm"]
Oct 30 08:06:56.809896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:56.809860    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:57.689305 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:57.689270    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" path="/var/lib/kubelet/pods/11e1dc2d-08cd-4754-a629-4edc43dc612c/volumes"
Oct 30 08:06:58.345986 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:58.345945    6738 patch_prober.go:29] interesting pod/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 08:06:58.346424 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:58.346009    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 08:06:58.603097 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:06:58.602999    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:00.129246 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.129211    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 08:07:00.186235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.186199    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\" (UniqueName: \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\") pod \"d7ab6451-b0af-4c56-a047-869b89e046c4\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") "
Oct 30 08:07:00.186446 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.186335    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\") pod \"d7ab6451-b0af-4c56-a047-869b89e046c4\" (UID: \"d7ab6451-b0af-4c56-a047-869b89e046c4\") "
Oct 30 08:07:00.186844 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.186813    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume" pod "d7ab6451-b0af-4c56-a047-869b89e046c4" (UID: "d7ab6451-b0af-4c56-a047-869b89e046c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:00.188509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.188479    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv" pod "d7ab6451-b0af-4c56-a047-869b89e046c4" (UID: "d7ab6451-b0af-4c56-a047-869b89e046c4"). InnerVolumeSpecName "kube-api-access-bvlxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:00.288445 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.288409    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvlxv\" (UniqueName: \"kubernetes.io/projected/d7ab6451-b0af-4c56-a047-869b89e046c4-kube-api-access-bvlxv\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.288445 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.288442    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ab6451-b0af-4c56-a047-869b89e046c4-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.466653 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.466611    6738 generic.go:356] "Generic (PLEG): container finished" podID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerID="8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef" exitCode=0
Oct 30 08:07:00.466872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.466658    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"
Oct 30 08:07:00.466872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.466658    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" event={"ID":"d7ab6451-b0af-4c56-a047-869b89e046c4","Type":"ContainerDied","Data":"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"}
Oct 30 08:07:00.466872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.466783    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc" event={"ID":"d7ab6451-b0af-4c56-a047-869b89e046c4","Type":"ContainerDied","Data":"590c640a01fc848fb04e643ce8d79c6bc2a3b84537ac21d2b5765a1d03d5490b"}
Oct 30 08:07:00.466872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.466815    6738 scope.go:122] "RemoveContainer" containerID="8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"
Oct 30 08:07:00.478499 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.478471    6738 scope.go:122] "RemoveContainer" containerID="8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"
Oct 30 08:07:00.478845 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:00.478826    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef\": container with ID starting with 8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef not found: ID does not exist" containerID="8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"
Oct 30 08:07:00.478907 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.478855    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef"} err="failed to get container status \"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef\": rpc error: code = NotFound desc = could not find container \"8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef\": container with ID starting with 8806466ebbfdd97646e140e398d94883eee345147a4efbd64d982b8ffed499ef not found: ID does not exist"
Oct 30 08:07:00.521046 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.521012    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 08:07:00.532811 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.532775    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:00.533871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:00.533838    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b5998f-6bc677769f-6spxc"]
Oct 30 08:07:01.674577 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.674548    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 08:07:01.689272 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.689188    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" path="/var/lib/kubelet/pods/d7ab6451-b0af-4c56-a047-869b89e046c4/volumes"
Oct 30 08:07:01.704354 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704317    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.704545 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704428    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.704545 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704460    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.704545 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704482    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.704842 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704629    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.704842 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704724    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:01.704995 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.704975    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.705068 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705040    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.705138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705066    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:01.705138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705096    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.705264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705141    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\") pod \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\" (UID: \"53bb06cf-e769-490e-87fd-4b2f4c04f42b\") "
Oct 30 08:07:01.705264 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705234    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:01.705452 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705428    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:01.705696 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705674    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.705821 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705715    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.705821 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705731    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.705821 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.705743    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.706445 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.706423    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:01.706926 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.706905    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:01.707194 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.707176    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "kube-api-access-5pbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:01.707327 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.707263    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:07:01.707652 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.707630    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token" pod "53bb06cf-e769-490e-87fd-4b2f4c04f42b" (UID: "53bb06cf-e769-490e-87fd-4b2f4c04f42b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:01.807030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.806980    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbh9\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-kube-api-access-5pbh9\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.807030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.807022    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.807030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.807032    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.807030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.807041    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:01.807399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:01.807049    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bb06cf-e769-490e-87fd-4b2f4c04f42b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:02.361067 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.361026    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 08:07:02.424918 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.424878    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 08:07:02.487357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.487322    6738 generic.go:356] "Generic (PLEG): container finished" podID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerID="3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b" exitCode=0
Oct 30 08:07:02.487357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.487357    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"
Oct 30 08:07:02.487713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.487389    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerDied","Data":"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"}
Oct 30 08:07:02.487713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.487456    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6" event={"ID":"53bb06cf-e769-490e-87fd-4b2f4c04f42b","Type":"ContainerDied","Data":"19af91c1d783b0cda6719bd09679420488ae3706237c4e75359618ff35f58160"}
Oct 30 08:07:02.487713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.487487    6738 scope.go:122] "RemoveContainer" containerID="3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"
Oct 30 08:07:02.502599 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.502571    6738 scope.go:122] "RemoveContainer" containerID="8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"
Oct 30 08:07:02.513017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.512990    6738 scope.go:122] "RemoveContainer" containerID="3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"
Oct 30 08:07:02.513419 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:02.513392    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b\": container with ID starting with 3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b not found: ID does not exist" containerID="3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"
Oct 30 08:07:02.513555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.513435    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b"} err="failed to get container status \"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b\": rpc error: code = NotFound desc = could not find container \"3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b\": container with ID starting with 3f61f0a5b5b630850429deda71604f03e5d284f47a4c84ca0c4b87c2b9ee329b not found: ID does not exist"
Oct 30 08:07:02.513555 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.513467    6738 scope.go:122] "RemoveContainer" containerID="8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"
Oct 30 08:07:02.513799 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:02.513779    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a\": container with ID starting with 8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a not found: ID does not exist" containerID="8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"
Oct 30 08:07:02.513858 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.513809    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a"} err="failed to get container status \"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a\": rpc error: code = NotFound desc = could not find container \"8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a\": container with ID starting with 8d04ccdfe69b290d5c1ce4d02fd7b96cd89c7a0893cc295f002adcd3c98b538a not found: ID does not exist"
Oct 30 08:07:02.556127 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.556087    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:02.575611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.575572    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 08:07:02.579489 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.579454    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-14271d-openshift-default-5bdc88cc59-4jfr6"]
Oct 30 08:07:02.683475 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:02.683441    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:03.689078 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:03.689040    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" path="/var/lib/kubelet/pods/53bb06cf-e769-490e-87fd-4b2f4c04f42b/volumes"
Oct 30 08:07:04.449189 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.449156    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:04.720666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.720635    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 08:07:04.743523 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.743495    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 08:07:04.842323 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842247    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\" (UniqueName: \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\") pod \"ac1d3bca-6e3b-448f-a8fc-48e635265401\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") "
Oct 30 08:07:04.842509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842368    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\") pod \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") "
Oct 30 08:07:04.842509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842406    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\" (UniqueName: \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\") pod \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\" (UID: \"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763\") "
Oct 30 08:07:04.842509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842441    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\") pod \"ac1d3bca-6e3b-448f-a8fc-48e635265401\" (UID: \"ac1d3bca-6e3b-448f-a8fc-48e635265401\") "
Oct 30 08:07:04.842856 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842831    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume" pod "8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" (UID: "8a3d9883-65cb-4fd5-aa1d-cf514f1ab763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:04.842943 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.842852    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume" pod "ac1d3bca-6e3b-448f-a8fc-48e635265401" (UID: "ac1d3bca-6e3b-448f-a8fc-48e635265401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:04.845119 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.845087    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v" pod "8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" (UID: "8a3d9883-65cb-4fd5-aa1d-cf514f1ab763"). InnerVolumeSpecName "kube-api-access-p478v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:04.845242 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.845096    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q" pod "ac1d3bca-6e3b-448f-a8fc-48e635265401" (UID: "ac1d3bca-6e3b-448f-a8fc-48e635265401"). InnerVolumeSpecName "kube-api-access-p6k7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:04.943729 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.943629    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1d3bca-6e3b-448f-a8fc-48e635265401-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:04.943729 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.943665    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k7q\" (UniqueName: \"kubernetes.io/projected/ac1d3bca-6e3b-448f-a8fc-48e635265401-kube-api-access-p6k7q\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:04.943729 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.943678    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:04.943729 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:04.943689    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p478v\" (UniqueName: \"kubernetes.io/projected/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763-kube-api-access-p478v\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:05.519922 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.519883    6738 generic.go:356] "Generic (PLEG): container finished" podID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerID="abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af" exitCode=0
Oct 30 08:07:05.520131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.519930    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"
Oct 30 08:07:05.520131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.519969    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" event={"ID":"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763","Type":"ContainerDied","Data":"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"}
Oct 30 08:07:05.520131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.520036    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8" event={"ID":"8a3d9883-65cb-4fd5-aa1d-cf514f1ab763","Type":"ContainerDied","Data":"88b4fd44247d9bf53572a72b6288b6134be0b13639a543880adbe9cbb93ab72a"}
Oct 30 08:07:05.520131 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.520068    6738 scope.go:122] "RemoveContainer" containerID="abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"
Oct 30 08:07:05.521733 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.521708    6738 generic.go:356] "Generic (PLEG): container finished" podID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerID="f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4" exitCode=0
Oct 30 08:07:05.521850 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.521803    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"
Oct 30 08:07:05.521850 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.521822    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" event={"ID":"ac1d3bca-6e3b-448f-a8fc-48e635265401","Type":"ContainerDied","Data":"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"}
Oct 30 08:07:05.521938 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.521867    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz" event={"ID":"ac1d3bca-6e3b-448f-a8fc-48e635265401","Type":"ContainerDied","Data":"67df1517a961539e62ab87950e2031d98ee7d9e7b765855591251f5d910f35bb"}
Oct 30 08:07:05.530790 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.530762    6738 scope.go:122] "RemoveContainer" containerID="abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"
Oct 30 08:07:05.531196 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:05.531099    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af\": container with ID starting with abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af not found: ID does not exist" containerID="abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"
Oct 30 08:07:05.531196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.531127    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af"} err="failed to get container status \"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af\": rpc error: code = NotFound desc = could not find container \"abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af\": container with ID starting with abc295f2af418af213d876b5d94d6f17cd1b9ea3222beaa0cdf81863ceb486af not found: ID does not exist"
Oct 30 08:07:05.531196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.531145    6738 scope.go:122] "RemoveContainer" containerID="f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"
Oct 30 08:07:05.541872 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.541854    6738 scope.go:122] "RemoveContainer" containerID="f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"
Oct 30 08:07:05.542211 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:05.542182    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4\": container with ID starting with f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4 not found: ID does not exist" containerID="f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"
Oct 30 08:07:05.542372 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.542216    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4"} err="failed to get container status \"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4\": rpc error: code = NotFound desc = could not find container \"f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4\": container with ID starting with f4c0498d83ef886f5e2a599a90b9c7c801ababb52f40de4fcf7db5a121ebe5d4 not found: ID does not exist"
Oct 30 08:07:05.577257 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.577221    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 08:07:05.589044 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.589009    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14271d-5ffd5fbdd8-88hg8"]
Oct 30 08:07:05.614006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.613973    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 08:07:05.616354 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.616329    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14271d-c4fc884d7-gj9pz"]
Oct 30 08:07:05.689764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.689684    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" path="/var/lib/kubelet/pods/8a3d9883-65cb-4fd5-aa1d-cf514f1ab763/volumes"
Oct 30 08:07:05.690160 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.690142    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" path="/var/lib/kubelet/pods/ac1d3bca-6e3b-448f-a8fc-48e635265401/volumes"
Oct 30 08:07:05.894050 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:05.894016    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:06.026132 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:06.026043    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:06.355470 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:06.355388    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:07.361935 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.361886    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing" containerID="cri-o://7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268" gracePeriod=25
Oct 30 08:07:07.426081 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.426029    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing" containerID="cri-o://b1e55d0c94669eefabb2c2eb72bb1844c343b347c8069bdfcb014e93486f48d8" gracePeriod=25
Oct 30 08:07:07.521312 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:07.521262    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:07:07.535109 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.535068    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 08:07:07.647844 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.647804    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:07.763650 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.763609    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:07.991066 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.990980    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"]
Oct 30 08:07:07.991438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991424    6738 cpu_manager.go:414] "containerMap: removing container" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991439    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991453    6738 cpu_manager.go:414] "containerMap: removing container" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991458    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991469    6738 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991475    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991486    6738 cpu_manager.go:414] "containerMap: removing container" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991491    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991499    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6a6b2f6e-493c-4df8-ad83-2dfd848ab723" containerName="revoke-keys"
Oct 30 08:07:07.991502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991504    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6b2f6e-493c-4df8-ad83-2dfd848ab723" containerName="revoke-keys"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991513    6738 cpu_manager.go:414] "containerMap: removing container" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991517    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991523    6738 cpu_manager.go:414] "containerMap: removing container" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991528    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991534    6738 cpu_manager.go:414] "containerMap: removing container" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991539    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991546    6738 cpu_manager.go:414] "containerMap: removing container" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991550    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991566    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991571    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991580    6738 cpu_manager.go:414] "containerMap: removing container" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991586    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991680    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991691    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991698    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="183fa159-dc30-4e2d-953e-e83ed0986556" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991706    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab6451-b0af-4c56-a047-869b89e046c4" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991712    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3d9883-65cb-4fd5-aa1d-cf514f1ab763" containerName="payload-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991718    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6b2f6e-493c-4df8-ad83-2dfd848ab723" containerName="revoke-keys"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991724    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bb06cf-e769-490e-87fd-4b2f4c04f42b" containerName="istio-proxy"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991731    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e276cef-464c-4b86-b80c-c26e265f57fb" containerName="payload-pre-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991737    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cdc8c8-6be6-46ef-9f1a-ff7e2c6505a1" containerName="payload-processing"
Oct 30 08:07:07.991901 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.991744    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1d3bca-6e3b-448f-a8fc-48e635265401" containerName="payload-pre-processing"
Oct 30 08:07:07.996797 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:07.996770    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.011925 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.011889    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:07:08.012287 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.012267    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:07:08.014403 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.014376    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"]
Oct 30 08:07:08.078438 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.078395    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.078664 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.078594    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.078664 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.078641    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd8n\" (UniqueName: \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.179841 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.179797    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.180073 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.179852    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd8n\" (UniqueName: \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.180073 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.179903    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.180609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.180587    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.182309 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.182274    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.206893 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.206853    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd8n\" (UniqueName: \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\") pod \"maas-api-revoke-sub-2a33320c-xlgm7\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.265096 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.265003    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:08.314115 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.314074    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:08.609088 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.608996    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:08.629921 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.629889    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"]
Oct 30 08:07:08.630661 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:07:08.630636    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d0c3f2_1adf_47d2_8f47_945ce0939972.slice/crio-82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5 WatchSource:0}: Error finding container 82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5: Status 404 returned error can't find the container with id 82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5
Oct 30 08:07:08.717980 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:08.717931    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:09.559693 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:09.559657    6738 generic.go:356] "Generic (PLEG): container finished" podID="20d0c3f2-1adf-47d2-8f47-945ce0939972" containerID="6314ab333e7123d109d13c02946249feab0ab74c08a93a2f0dd600e1f0ee7cec" exitCode=0
Oct 30 08:07:09.559865 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:09.559741    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7" event={"ID":"20d0c3f2-1adf-47d2-8f47-945ce0939972","Type":"ContainerDied","Data":"6314ab333e7123d109d13c02946249feab0ab74c08a93a2f0dd600e1f0ee7cec"}
Oct 30 08:07:09.559865 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:09.559798    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7" event={"ID":"20d0c3f2-1adf-47d2-8f47-945ce0939972","Type":"ContainerStarted","Data":"82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5"}
Oct 30 08:07:10.433630 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.433590    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:10.713605 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.713576    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:10.814220 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.814183    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\" (UniqueName: \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\") pod \"20d0c3f2-1adf-47d2-8f47-945ce0939972\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") "
Oct 30 08:07:10.814457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.814265    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\") pod \"20d0c3f2-1adf-47d2-8f47-945ce0939972\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") "
Oct 30 08:07:10.814457 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.814371    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\") pod \"20d0c3f2-1adf-47d2-8f47-945ce0939972\" (UID: \"20d0c3f2-1adf-47d2-8f47-945ce0939972\") "
Oct 30 08:07:10.814639 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.814610    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca" pod "20d0c3f2-1adf-47d2-8f47-945ce0939972" (UID: "20d0c3f2-1adf-47d2-8f47-945ce0939972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:10.814779 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.814765    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:10.816504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.816479    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca" pod "20d0c3f2-1adf-47d2-8f47-945ce0939972" (UID: "20d0c3f2-1adf-47d2-8f47-945ce0939972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:10.816623 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.816480    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n" pod "20d0c3f2-1adf-47d2-8f47-945ce0939972" (UID: "20d0c3f2-1adf-47d2-8f47-945ce0939972"). InnerVolumeSpecName "kube-api-access-hpd8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:10.916319 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.916264    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpd8n\" (UniqueName: \"kubernetes.io/projected/20d0c3f2-1adf-47d2-8f47-945ce0939972-kube-api-access-hpd8n\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:10.916319 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:10.916304    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d0c3f2-1adf-47d2-8f47-945ce0939972-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.574534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.574500    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7"
Oct 30 08:07:11.574972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.574531    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a33320c-xlgm7" event={"ID":"20d0c3f2-1adf-47d2-8f47-945ce0939972","Type":"ContainerDied","Data":"82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5"}
Oct 30 08:07:11.574972 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.574623    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82da96f515d29e73c04b5ca821e37ebfcc253bea5659e22f12ebf130a46991c5"
Oct 30 08:07:11.588018 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.587966    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" containerID="cri-o://3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53" gracePeriod=45
Oct 30 08:07:11.712307 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.712247    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 08:07:11.848114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.848083    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 08:07:11.921805 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.921772    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 08:07:11.922002 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.921968    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" podUID="341a8986-9869-4907-8bcf-b66e3dbd71be" containerName="maas-api" containerID="cri-o://07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907" gracePeriod=30
Oct 30 08:07:11.928836 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.928805    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\" (UniqueName: \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.929024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.928893    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.929024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.928955    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.929024 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929008    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\" (UniqueName: \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.929230 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929051    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.929230 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929120    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\") pod \"db64ba02-617e-46e4-997c-ef93cd74d222\" (UID: \"db64ba02-617e-46e4-997c-ef93cd74d222\") "
Oct 30 08:07:11.930665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929528    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.930665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929709    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.930665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929710    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.930665 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.929964    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.932553 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.932522    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.933148 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.933124    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "kube-api-access-pq5k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:11.933248 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:11.933197    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs" pod "db64ba02-617e-46e4-997c-ef93cd74d222" (UID: "db64ba02-617e-46e4-997c-ef93cd74d222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:12.006176 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.006138    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:12.030380 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.030341    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.030380 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.030371    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db64ba02-617e-46e4-997c-ef93cd74d222-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.030380 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.030381    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.030380 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.030389    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5k5\" (UniqueName: \"kubernetes.io/projected/db64ba02-617e-46e4-997c-ef93cd74d222-kube-api-access-pq5k5\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.030766 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.030398    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db64ba02-617e-46e4-997c-ef93cd74d222-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.185251 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.185226    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 08:07:12.333488 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.333450    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\" (UniqueName: \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\") pod \"341a8986-9869-4907-8bcf-b66e3dbd71be\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") "
Oct 30 08:07:12.333661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.333504    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\") pod \"341a8986-9869-4907-8bcf-b66e3dbd71be\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") "
Oct 30 08:07:12.333661 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.333608    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\") pod \"341a8986-9869-4907-8bcf-b66e3dbd71be\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") "
Oct 30 08:07:12.333795 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.333768    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\") pod \"341a8986-9869-4907-8bcf-b66e3dbd71be\" (UID: \"341a8986-9869-4907-8bcf-b66e3dbd71be\") "
Oct 30 08:07:12.333968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.333941    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca" pod "341a8986-9869-4907-8bcf-b66e3dbd71be" (UID: "341a8986-9869-4907-8bcf-b66e3dbd71be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:12.334183 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.334161    6738 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341a8986-9869-4907-8bcf-b66e3dbd71be-openshift-service-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.335717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.335696    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls" pod "341a8986-9869-4907-8bcf-b66e3dbd71be" (UID: "341a8986-9869-4907-8bcf-b66e3dbd71be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:12.335873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.335727    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs" pod "341a8986-9869-4907-8bcf-b66e3dbd71be" (UID: "341a8986-9869-4907-8bcf-b66e3dbd71be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:12.335873 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.335739    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j" pod "341a8986-9869-4907-8bcf-b66e3dbd71be" (UID: "341a8986-9869-4907-8bcf-b66e3dbd71be"). InnerVolumeSpecName "kube-api-access-k7c4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:12.437219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.435954    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c4j\" (UniqueName: \"kubernetes.io/projected/341a8986-9869-4907-8bcf-b66e3dbd71be-kube-api-access-k7c4j\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.437219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.435999    6738 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-metrics-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.437219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.436014    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341a8986-9869-4907-8bcf-b66e3dbd71be-maas-api-tls\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.535629 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.535584    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing" containerID="cri-o://bd24311cf41cf9f4c93312d53da0374aafe8b30008d7dd6675b4f7e7decade20" gracePeriod=25
Oct 30 08:07:12.583002 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.582969    6738 generic.go:356] "Generic (PLEG): container finished" podID="db64ba02-617e-46e4-997c-ef93cd74d222" containerID="3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53" exitCode=0
Oct 30 08:07:12.583497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.583008    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"
Oct 30 08:07:12.583497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.583053    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" event={"ID":"db64ba02-617e-46e4-997c-ef93cd74d222","Type":"ContainerDied","Data":"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"}
Oct 30 08:07:12.583497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.583096    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf" event={"ID":"db64ba02-617e-46e4-997c-ef93cd74d222","Type":"ContainerDied","Data":"c5bc8e9b6611dae1e21599c8722cb9c7a6e498f8239aa5cfc38d889d3fe845fa"}
Oct 30 08:07:12.583497 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.583114    6738 scope.go:122] "RemoveContainer" containerID="3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"
Oct 30 08:07:12.584958 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.584887    6738 generic.go:356] "Generic (PLEG): container finished" podID="341a8986-9869-4907-8bcf-b66e3dbd71be" containerID="07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907" exitCode=0
Oct 30 08:07:12.584958 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.584919    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" event={"ID":"341a8986-9869-4907-8bcf-b66e3dbd71be","Type":"ContainerDied","Data":"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"}
Oct 30 08:07:12.584958 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.584952    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"
Oct 30 08:07:12.585164 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.584963    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25" event={"ID":"341a8986-9869-4907-8bcf-b66e3dbd71be","Type":"ContainerDied","Data":"6fe74afa6bcfb733b0860b8e22aaa4566ee97304224f8b1247c78e67e02d5874"}
Oct 30 08:07:12.593674 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.593622    6738 scope.go:122] "RemoveContainer" containerID="3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"
Oct 30 08:07:12.593982 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:12.593956    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53\": container with ID starting with 3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53 not found: ID does not exist" containerID="3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"
Oct 30 08:07:12.594128 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.593988    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53"} err="failed to get container status \"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53\": rpc error: code = NotFound desc = could not find container \"3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53\": container with ID starting with 3494062c57ea4d26d4162951ba2bee3248461abf84c4e01966464cb70bbb6e53 not found: ID does not exist"
Oct 30 08:07:12.594128 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.594012    6738 scope.go:122] "RemoveContainer" containerID="07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"
Oct 30 08:07:12.603963 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.603945    6738 scope.go:122] "RemoveContainer" containerID="07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"
Oct 30 08:07:12.604267 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:12.604247    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907\": container with ID starting with 07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907 not found: ID does not exist" containerID="07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"
Oct 30 08:07:12.604346 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.604274    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907"} err="failed to get container status \"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907\": rpc error: code = NotFound desc = could not find container \"07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907\": container with ID starting with 07e7156e26f37cd5db0ab2152ff766fcec07c520cb3d8e8418bf6ba15c15e907 not found: ID does not exist"
Oct 30 08:07:12.638091 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.638054    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 08:07:12.649207 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.649163    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f5ceca-767dccd6c-h6x25"]
Oct 30 08:07:12.686807 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.686771    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 08:07:12.696996 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.696912    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-2-w0-06b26e-kserve-79b95d4b89-82fcf"]
Oct 30 08:07:12.741567 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.741502    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:12.898622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:12.898585    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:13.460702 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:13.460662    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:13.688520 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:13.688489    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341a8986-9869-4907-8bcf-b66e3dbd71be" path="/var/lib/kubelet/pods/341a8986-9869-4907-8bcf-b66e3dbd71be/volumes"
Oct 30 08:07:13.688942 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:13.688929    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" path="/var/lib/kubelet/pods/db64ba02-617e-46e4-997c-ef93cd74d222/volumes"
Oct 30 08:07:15.209951 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:15.204810    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:15.668416 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:15.668372    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:15.668636 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:15.668513    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:17.162689 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:17.162643    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:17.738645 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:17.738607    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:18.691211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:18.691175    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:19.266973 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:19.266935    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:19.381841 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:19.381796    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:20.074347 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:20.074311    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:20.523742 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:20.523703    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:20.680222 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:20.680179    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:20.933267 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:20.933220    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:21.389049 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.389006    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:21.552531 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.552489    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:21.668210 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668125    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"]
Oct 30 08:07:21.668869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668843    6738 cpu_manager.go:414] "containerMap: removing container" podUID="341a8986-9869-4907-8bcf-b66e3dbd71be" containerName="maas-api"
Oct 30 08:07:21.668869 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668870    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341a8986-9869-4907-8bcf-b66e3dbd71be" containerName="maas-api"
Oct 30 08:07:21.669042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668930    6738 cpu_manager.go:414] "containerMap: removing container" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main"
Oct 30 08:07:21.669042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668939    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main"
Oct 30 08:07:21.669042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668965    6738 cpu_manager.go:414] "containerMap: removing container" podUID="20d0c3f2-1adf-47d2-8f47-945ce0939972" containerName="revoke-keys"
Oct 30 08:07:21.669042 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.668974    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d0c3f2-1adf-47d2-8f47-945ce0939972" containerName="revoke-keys"
Oct 30 08:07:21.669193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.669068    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d0c3f2-1adf-47d2-8f47-945ce0939972" containerName="revoke-keys"
Oct 30 08:07:21.669193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.669078    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="341a8986-9869-4907-8bcf-b66e3dbd71be" containerName="maas-api"
Oct 30 08:07:21.669193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.669090    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1dc2d-08cd-4754-a629-4edc43dc612c" containerName="istio-proxy"
Oct 30 08:07:21.669193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.669097    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="db64ba02-617e-46e4-997c-ef93cd74d222" containerName="main"
Oct 30 08:07:21.674666 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.674639    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.689722 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.689692    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:07:21.692736 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.692701    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"]
Oct 30 08:07:21.699658 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.699621    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:07:21.699870 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.699681    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 08:07:21.699870 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.699753    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 08:07:21.838475 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.838437    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.838675 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.838485    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkn4\" (UniqueName: \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.838675 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.838602    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.939884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.939796    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.939884 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.939842    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkn4\" (UniqueName: \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.940101 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.939903    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.940546 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.940522    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.942364 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.942342    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.964236 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.964205    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkn4\" (UniqueName: \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:21.993107 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:21.993066    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:22.319014 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:22.318983    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"]
Oct 30 08:07:22.320088 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:07:22.320065    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f02c5da_2605_4780_8a1a_9499a5a19404.slice/crio-08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a WatchSource:0}: Error finding container 08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a: Status 404 returned error can't find the container with id 08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a
Oct 30 08:07:22.662914 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:22.662876    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw" event={"ID":"6f02c5da-2605-4780-8a1a-9499a5a19404","Type":"ContainerStarted","Data":"44dedafe415ff1670c7183806eb0624cc9d897327b0fc3600d3859efce76432c"}
Oct 30 08:07:22.663469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:22.662927    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw" event={"ID":"6f02c5da-2605-4780-8a1a-9499a5a19404","Type":"ContainerStarted","Data":"08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a"}
Oct 30 08:07:22.709580 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:22.709521    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw" podStartSLOduration=1.709506381 podStartE2EDuration="1.709506381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:22.707762726 +0000 UTC m=+4203.140526620" watchObservedRunningTime="2026-10-30 08:07:22.709506381 +0000 UTC m=+4203.142270293"
Oct 30 08:07:23.671380 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:23.671339    6738 generic.go:356] "Generic (PLEG): container finished" podID="6f02c5da-2605-4780-8a1a-9499a5a19404" containerID="44dedafe415ff1670c7183806eb0624cc9d897327b0fc3600d3859efce76432c" exitCode=0
Oct 30 08:07:23.671812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:23.671428    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw" event={"ID":"6f02c5da-2605-4780-8a1a-9499a5a19404","Type":"ContainerDied","Data":"44dedafe415ff1670c7183806eb0624cc9d897327b0fc3600d3859efce76432c"}
Oct 30 08:07:23.946150 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:23.946058    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:24.821162 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.821132    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:24.974374 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.974256    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\") pod \"6f02c5da-2605-4780-8a1a-9499a5a19404\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") "
Oct 30 08:07:24.974374 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.974371    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\") pod \"6f02c5da-2605-4780-8a1a-9499a5a19404\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") "
Oct 30 08:07:24.974583 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.974447    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\" (UniqueName: \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\") pod \"6f02c5da-2605-4780-8a1a-9499a5a19404\" (UID: \"6f02c5da-2605-4780-8a1a-9499a5a19404\") "
Oct 30 08:07:24.974755 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.974730    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca" pod "6f02c5da-2605-4780-8a1a-9499a5a19404" (UID: "6f02c5da-2605-4780-8a1a-9499a5a19404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:24.976516 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.976491    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca" pod "6f02c5da-2605-4780-8a1a-9499a5a19404" (UID: "6f02c5da-2605-4780-8a1a-9499a5a19404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:24.976626 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:24.976600    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4" pod "6f02c5da-2605-4780-8a1a-9499a5a19404" (UID: "6f02c5da-2605-4780-8a1a-9499a5a19404"). InnerVolumeSpecName "kube-api-access-pkkn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:25.075472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.075427    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:25.075472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.075462    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f02c5da-2605-4780-8a1a-9499a5a19404-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:25.075472 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.075475    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkn4\" (UniqueName: \"kubernetes.io/projected/6f02c5da-2605-4780-8a1a-9499a5a19404-kube-api-access-pkkn4\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:25.668386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.668337    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:25.668554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.668398    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:25.687887 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.687858    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"
Oct 30 08:07:25.689607 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.689573    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw" event={"ID":"6f02c5da-2605-4780-8a1a-9499a5a19404","Type":"ContainerDied","Data":"08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a"}
Oct 30 08:07:25.689769 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.689631    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08869eff264afcf748ab613f1490a2c29996b8f10c797a78b928a773c5a3094a"
Oct 30 08:07:25.765946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:25.765902    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:26.348966 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:26.348924    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 08:07:26.361360 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:26.361328    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 08:07:26.483805 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:26.483771    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:26.979110 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:26.979077    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:27.105591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.105548    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:27.338629 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.338535    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:27.700783 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.700752    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 08:07:27.704591 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.704559    6738 generic.go:356] "Generic (PLEG): container finished" podID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerID="b1e55d0c94669eefabb2c2eb72bb1844c343b347c8069bdfcb014e93486f48d8" exitCode=0
Oct 30 08:07:27.704764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.704623    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" event={"ID":"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33","Type":"ContainerDied","Data":"b1e55d0c94669eefabb2c2eb72bb1844c343b347c8069bdfcb014e93486f48d8"}
Oct 30 08:07:27.704764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.704680    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws" event={"ID":"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33","Type":"ContainerDied","Data":"b637c154e9a1835b4612cdcbb732abf8f88cb0151bdf809f408cf8b4295756ba"}
Oct 30 08:07:27.704764 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.704703    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b637c154e9a1835b4612cdcbb732abf8f88cb0151bdf809f408cf8b4295756ba"
Oct 30 08:07:27.706269 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706247    6738 generic.go:356] "Generic (PLEG): container finished" podID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerID="7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268" exitCode=0
Oct 30 08:07:27.706397 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706283    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"
Oct 30 08:07:27.706397 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706341    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" event={"ID":"7d426c3b-8ec2-4433-bc09-2a210b93d6b4","Type":"ContainerDied","Data":"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"}
Oct 30 08:07:27.706397 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706386    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g" event={"ID":"7d426c3b-8ec2-4433-bc09-2a210b93d6b4","Type":"ContainerDied","Data":"11e796a8c43d03dd2ca701cbd00ec0b9a98f1fbf778331aa7500083759ab142b"}
Oct 30 08:07:27.706524 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706417    6738 scope.go:122] "RemoveContainer" containerID="7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"
Oct 30 08:07:27.706906 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706882    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\" (UniqueName: \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\") pod \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") "
Oct 30 08:07:27.707032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.706931    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\") pod \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\" (UID: \"7d426c3b-8ec2-4433-bc09-2a210b93d6b4\") "
Oct 30 08:07:27.707369 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.707348    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume" pod "7d426c3b-8ec2-4433-bc09-2a210b93d6b4" (UID: "7d426c3b-8ec2-4433-bc09-2a210b93d6b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:27.709255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.709231    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr" pod "7d426c3b-8ec2-4433-bc09-2a210b93d6b4" (UID: "7d426c3b-8ec2-4433-bc09-2a210b93d6b4"). InnerVolumeSpecName "kube-api-access-6qslr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:27.717454 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.717430    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 08:07:27.718197 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.718178    6738 scope.go:122] "RemoveContainer" containerID="7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"
Oct 30 08:07:27.718559 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:27.718538    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268\": container with ID starting with 7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268 not found: ID does not exist" containerID="7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"
Oct 30 08:07:27.718620 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.718568    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268"} err="failed to get container status \"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268\": rpc error: code = NotFound desc = could not find container \"7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268\": container with ID starting with 7b4c98c8b6456445e8e8947ba4cbb3920c583112563a23d0669dcc021c808268 not found: ID does not exist"
Oct 30 08:07:27.738350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.738312    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:27.809036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.808141    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\") pod \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") "
Oct 30 08:07:27.809036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.808843    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\" (UniqueName: \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\") pod \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\" (UID: \"5ed47358-4b3b-4aa6-bbed-c7bb7235ac33\") "
Oct 30 08:07:27.809036 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.808873    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume" pod "5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" (UID: "5ed47358-4b3b-4aa6-bbed-c7bb7235ac33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:27.809998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.809855    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:27.809998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.809888    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qslr\" (UniqueName: \"kubernetes.io/projected/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-kube-api-access-6qslr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:27.809998 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.809902    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d426c3b-8ec2-4433-bc09-2a210b93d6b4-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:27.813199 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.813163    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr" pod "5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" (UID: "5ed47358-4b3b-4aa6-bbed-c7bb7235ac33"). InnerVolumeSpecName "kube-api-access-jrgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:27.844008 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.843973    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:27.910850 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:27.910761    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrgtr\" (UniqueName: \"kubernetes.io/projected/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33-kube-api-access-jrgtr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:28.061658 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:28.061624    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 08:07:28.072278 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:28.072241    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-06b26e-56b64dc668-tn22g"]
Oct 30 08:07:28.715232 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:28.715150    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"
Oct 30 08:07:28.777116 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:28.777075    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 08:07:28.789238 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:28.789195    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-06b26e-7d7c4c5667-g58ws"]
Oct 30 08:07:29.659063 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:29.659024    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:29.689992 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:29.689957    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" path="/var/lib/kubelet/pods/5ed47358-4b3b-4aa6-bbed-c7bb7235ac33/volumes"
Oct 30 08:07:29.690905 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:29.690877    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" path="/var/lib/kubelet/pods/7d426c3b-8ec2-4433-bc09-2a210b93d6b4/volumes"
Oct 30 08:07:31.458732 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.458695    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:31.733089 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.733008    6738 scope.go:122] "RemoveContainer" containerID="14ab2cd48e176d8bb88704e7625fca77e6580ea26c8d5d3799ed72a568b94245"
Oct 30 08:07:31.744008 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.743986    6738 scope.go:122] "RemoveContainer" containerID="a7e0f0f9db05f387546f19578684147e3cb88aaf8b2c9e1a78e504b3ebb79dea"
Oct 30 08:07:31.753271 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.753247    6738 scope.go:122] "RemoveContainer" containerID="b1e55d0c94669eefabb2c2eb72bb1844c343b347c8069bdfcb014e93486f48d8"
Oct 30 08:07:31.763535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.763510    6738 scope.go:122] "RemoveContainer" containerID="ef29b1531b3e488295f4985b101892f9a6518628b68a759f761ca2848993f6e6"
Oct 30 08:07:31.773587 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.773470    6738 scope.go:122] "RemoveContainer" containerID="570cda668506b1b4fdcf17d595f33d9a63141cf8e05920825a29dc14d645fc54"
Oct 30 08:07:31.785819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.785794    6738 scope.go:122] "RemoveContainer" containerID="995b787c28a2634c47369284526f05260646df79712b98c5ad9b2f703cbea545"
Oct 30 08:07:31.794930 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.794909    6738 scope.go:122] "RemoveContainer" containerID="3831dcd58f59481ddb82088915adb681bb006cbb6285ace4d2229c4e3b2f3226"
Oct 30 08:07:31.804305 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.804271    6738 scope.go:122] "RemoveContainer" containerID="b07d910d98f3d514e642866663c6386b7a7701ac30f4a01ffee00f1da59cad61"
Oct 30 08:07:31.813090 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.813066    6738 scope.go:122] "RemoveContainer" containerID="02023c95400dbad8d4c4fe9616a16712239b0daf90df5fc0281957c0a7e9d21f"
Oct 30 08:07:31.822085 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:31.822065    6738 scope.go:122] "RemoveContainer" containerID="cba6fa2fc7fa746688b987676f4c33189228b2fda8fc9f353f178878f2680a25"
Oct 30 08:07:32.406502 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.406457    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:32.537216 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.537176    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:32.746446 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.746410    6738 generic.go:356] "Generic (PLEG): container finished" podID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerID="bd24311cf41cf9f4c93312d53da0374aafe8b30008d7dd6675b4f7e7decade20" exitCode=0
Oct 30 08:07:32.746639 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.746463    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" event={"ID":"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2","Type":"ContainerDied","Data":"bd24311cf41cf9f4c93312d53da0374aafe8b30008d7dd6675b4f7e7decade20"}
Oct 30 08:07:32.788458 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.788429    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 08:07:32.863353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.862550    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\" (UniqueName: \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\") pod \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") "
Oct 30 08:07:32.863353 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.862636    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\") pod \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\" (UID: \"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2\") "
Oct 30 08:07:32.863723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.863406    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume" pod "ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" (UID: "ced3e4b8-8cd1-42e3-8f14-826ac749d4d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:32.877095 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.877029    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr" pod "ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" (UID: "ced3e4b8-8cd1-42e3-8f14-826ac749d4d2"). InnerVolumeSpecName "kube-api-access-mrwrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:32.964333 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.964212    6738 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-plugins-config-volume\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:32.964333 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:32.964248    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwrr\" (UniqueName: \"kubernetes.io/projected/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2-kube-api-access-mrwrr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:33.476875 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.476836    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:33.756557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.756469    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn" event={"ID":"ced3e4b8-8cd1-42e3-8f14-826ac749d4d2","Type":"ContainerDied","Data":"055dfd703929f3a1576cb8acde0a6ab7a52b03dfe4fe725491e5b09fbaf8849b"}
Oct 30 08:07:33.756557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.756502    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"
Oct 30 08:07:33.756557 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.756539    6738 scope.go:122] "RemoveContainer" containerID="bd24311cf41cf9f4c93312d53da0374aafe8b30008d7dd6675b4f7e7decade20"
Oct 30 08:07:33.817655 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.817616    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 08:07:33.832017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.831989    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:33.836146 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:33.836119    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f5ceca-5d9788985f-6q7pn"]
Oct 30 08:07:34.015251 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:34.015158    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:35.229751 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.229718    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"]
Oct 30 08:07:35.230161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230113    6738 cpu_manager.go:414] "containerMap: removing container" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing"
Oct 30 08:07:35.230161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230124    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing"
Oct 30 08:07:35.230161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230145    6738 cpu_manager.go:414] "containerMap: removing container" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing"
Oct 30 08:07:35.230161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230150    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing"
Oct 30 08:07:35.230161 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230164    6738 cpu_manager.go:414] "containerMap: removing container" podUID="6f02c5da-2605-4780-8a1a-9499a5a19404" containerName="revoke-keys"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230169    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f02c5da-2605-4780-8a1a-9499a5a19404" containerName="revoke-keys"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230187    6738 cpu_manager.go:414] "containerMap: removing container" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230192    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230244    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d426c3b-8ec2-4433-bc09-2a210b93d6b4" containerName="payload-processing"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230253    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" containerName="payload-pre-processing"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230263    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed47358-4b3b-4aa6-bbed-c7bb7235ac33" containerName="payload-pre-processing"
Oct 30 08:07:35.230386 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.230269    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f02c5da-2605-4780-8a1a-9499a5a19404" containerName="revoke-keys"
Oct 30 08:07:35.234968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.234950    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.275845 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.275815    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 08:07:35.276064 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.275873    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 08:07:35.286816 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.286786    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.286894 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.286843    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv59\" (UniqueName: \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.286971 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.286951    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.288592 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.288568    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:07:35.288741 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.288690    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:07:35.293337 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.293308    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"]
Oct 30 08:07:35.388488 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.388436    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.388674 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.388495    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv59\" (UniqueName: \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.388674 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.388527    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.389142 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.389116    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.390944 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.390923    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.412986 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.412955    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv59\" (UniqueName: \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\") pod \"maas-api-revoke-sub-ad14e187-2xxpk\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.547758 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.547661    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:35.627730 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.627693    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:35.688420 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.688378    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced3e4b8-8cd1-42e3-8f14-826ac749d4d2" path="/var/lib/kubelet/pods/ced3e4b8-8cd1-42e3-8f14-826ac749d4d2/volumes"
Oct 30 08:07:35.851992 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:35.851960    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"]
Oct 30 08:07:35.853196 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:07:35.853162    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode685b84c_5da3_4d34_8d40_446bb69450c2.slice/crio-a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193 WatchSource:0}: Error finding container a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193: Status 404 returned error can't find the container with id a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193
Oct 30 08:07:36.598921 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:36.598827    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:36.780001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:36.779963    6738 generic.go:356] "Generic (PLEG): container finished" podID="e685b84c-5da3-4d34-8d40-446bb69450c2" containerID="996573b0758efe680e29b2b5bf6f906fdf71d1520a939d4353fc5e76019d721b" exitCode=0
Oct 30 08:07:36.780196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:36.780048    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk" event={"ID":"e685b84c-5da3-4d34-8d40-446bb69450c2","Type":"ContainerDied","Data":"996573b0758efe680e29b2b5bf6f906fdf71d1520a939d4353fc5e76019d721b"}
Oct 30 08:07:36.780196 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:36.780102    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk" event={"ID":"e685b84c-5da3-4d34-8d40-446bb69450c2","Type":"ContainerStarted","Data":"a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193"}
Oct 30 08:07:37.518559 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:37.518522    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:07:37.921963 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:37.921936    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:38.016266 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.016219    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\" (UniqueName: \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\") pod \"e685b84c-5da3-4d34-8d40-446bb69450c2\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") "
Oct 30 08:07:38.016513 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.016348    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\") pod \"e685b84c-5da3-4d34-8d40-446bb69450c2\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") "
Oct 30 08:07:38.016594 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.016516    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\") pod \"e685b84c-5da3-4d34-8d40-446bb69450c2\" (UID: \"e685b84c-5da3-4d34-8d40-446bb69450c2\") "
Oct 30 08:07:38.016770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.016739    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca" pod "e685b84c-5da3-4d34-8d40-446bb69450c2" (UID: "e685b84c-5da3-4d34-8d40-446bb69450c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:38.016851 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.016837    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-openshift-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:38.018469 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.018446    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59" pod "e685b84c-5da3-4d34-8d40-446bb69450c2" (UID: "e685b84c-5da3-4d34-8d40-446bb69450c2"). InnerVolumeSpecName "kube-api-access-cfv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:38.018552 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.018527    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca" pod "e685b84c-5da3-4d34-8d40-446bb69450c2" (UID: "e685b84c-5da3-4d34-8d40-446bb69450c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:38.118117 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.118030    6738 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e685b84c-5da3-4d34-8d40-446bb69450c2-maas-api-xks-ca\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:38.118117 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.118061    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfv59\" (UniqueName: \"kubernetes.io/projected/e685b84c-5da3-4d34-8d40-446bb69450c2-kube-api-access-cfv59\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:38.793111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.793074    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk"
Oct 30 08:07:38.793276 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.793103    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad14e187-2xxpk" event={"ID":"e685b84c-5da3-4d34-8d40-446bb69450c2","Type":"ContainerDied","Data":"a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193"}
Oct 30 08:07:38.793276 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:38.793160    6738 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6d8747f752bd18a2f8c3bb593856c8859785439949cb61af8a0c44a9a10f193"
Oct 30 08:07:41.370366 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.370322    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main" containerID="cri-o://691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05" gracePeriod=45
Oct 30 08:07:41.386819 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.386764    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" containerID="cri-o://fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1" gracePeriod=45
Oct 30 08:07:41.410695 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.410656    6738 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused"
Oct 30 08:07:41.640352 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.640324    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 08:07:41.662240 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.662211    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 08:07:41.758254 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758217    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.758554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758359    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758402    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.758554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758435    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.758554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758461    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758554 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758487    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758559    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\" (UniqueName: \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758628    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\" (UniqueName: \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758685    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758730    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758767    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\") pod \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\" (UID: \"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb\") "
Oct 30 08:07:41.758933 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.758801    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\") pod \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\" (UID: \"9f41d75b-e6a7-443e-8608-d1fb7e161e20\") "
Oct 30 08:07:41.760017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.759870    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.760165 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.760037    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.760429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.760393    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.760538 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.760455    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.760538 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.760504    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.760752 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.760711    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.762571 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.762538    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:41.762687 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.762661    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "kube-api-access-m4gjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:41.763047 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.763018    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.763173 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.763064    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm" pod "9f41d75b-e6a7-443e-8608-d1fb7e161e20" (UID: "9f41d75b-e6a7-443e-8608-d1fb7e161e20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.763874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.763847    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:41.764509 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.764487    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp" pod "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" (UID: "2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb"). InnerVolumeSpecName "kube-api-access-94blp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:41.817817 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.817784    6738 generic.go:356] "Generic (PLEG): container finished" podID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerID="fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1" exitCode=0
Oct 30 08:07:41.818028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.817856    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"
Oct 30 08:07:41.818028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.817858    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" event={"ID":"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb","Type":"ContainerDied","Data":"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"}
Oct 30 08:07:41.818028 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.817986    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv" event={"ID":"2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb","Type":"ContainerDied","Data":"4ffc885bb6de5a55273e594959495557cdfb71cf4f27fd34e123dceea55d7479"}
Oct 30 08:07:41.818182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.818033    6738 scope.go:122] "RemoveContainer" containerID="fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"
Oct 30 08:07:41.819717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.819571    6738 generic.go:356] "Generic (PLEG): container finished" podID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerID="691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05" exitCode=0
Oct 30 08:07:41.819717 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.819628    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"
Oct 30 08:07:41.819871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.819736    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" event={"ID":"9f41d75b-e6a7-443e-8608-d1fb7e161e20","Type":"ContainerDied","Data":"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"}
Oct 30 08:07:41.819871 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.819806    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq" event={"ID":"9f41d75b-e6a7-443e-8608-d1fb7e161e20","Type":"ContainerDied","Data":"2915a1db60e466e62861de123e98dc86e6f686281944e933bba7d2cd0d65965f"}
Oct 30 08:07:41.829724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.829555    6738 scope.go:122] "RemoveContainer" containerID="fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"
Oct 30 08:07:41.829914 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:41.829892    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1\": container with ID starting with fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1 not found: ID does not exist" containerID="fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"
Oct 30 08:07:41.829991 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.829924    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1"} err="failed to get container status \"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1\": rpc error: code = NotFound desc = could not find container \"fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1\": container with ID starting with fd588743c5bc3bcc9c7b926493c4d86cade7b0ce6a892a4f990f6f84b289b2c1 not found: ID does not exist"
Oct 30 08:07:41.829991 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.829943    6738 scope.go:122] "RemoveContainer" containerID="691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"
Oct 30 08:07:41.840184 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.840163    6738 scope.go:122] "RemoveContainer" containerID="691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"
Oct 30 08:07:41.840645 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:07:41.840624    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05\": container with ID starting with 691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05 not found: ID does not exist" containerID="691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"
Oct 30 08:07:41.840738 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.840658    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05"} err="failed to get container status \"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05\": rpc error: code = NotFound desc = could not find container \"691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05\": container with ID starting with 691b9cb6043bf6e0d5c7a388fe143633f265eac63e73598fedc1b0384768ed05 not found: ID does not exist"
Oct 30 08:07:41.860802 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860765    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94blp\" (UniqueName: \"kubernetes.io/projected/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-kube-api-access-94blp\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.860802 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860802    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gjr\" (UniqueName: \"kubernetes.io/projected/9f41d75b-e6a7-443e-8608-d1fb7e161e20-kube-api-access-m4gjr\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860816    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860830    6738 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-dshm\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860842    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860854    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860866    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860877    6738 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-home\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860891    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f41d75b-e6a7-443e-8608-d1fb7e161e20-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860902    6738 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f41d75b-e6a7-443e-8608-d1fb7e161e20-model-cache\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860913    6738 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tls-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.861034 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.860924    6738 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb-tmp-dir\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.874758 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.874686    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 08:07:41.877826 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.877800    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-w3-2eda2c-kserve-7749b48c75-gdftq"]
Oct 30 08:07:41.914525 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.914488    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 08:07:41.920192 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:41.920161    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/premium-w3-2eda2c-kserve-7f964dbd4-4djcv"]
Oct 30 08:07:42.566713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:42.566683    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:43.688778 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:43.688744    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" path="/var/lib/kubelet/pods/2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb/volumes"
Oct 30 08:07:43.689219 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:43.689203    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" path="/var/lib/kubelet/pods/9f41d75b-e6a7-443e-8608-d1fb7e161e20/volumes"
Oct 30 08:07:48.030564 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:48.030527    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"]
Oct 30 08:07:48.043177 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:48.043118    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"]
Oct 30 08:07:48.046105 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:48.046066    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4818406e-gjms9"]
Oct 30 08:07:48.047994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:48.047971    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1600e8f-pg9gd"]
Oct 30 08:07:49.688037 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:49.688004    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0740f920-de80-4520-8801-3a4babbcc199" path="/var/lib/kubelet/pods/0740f920-de80-4520-8801-3a4babbcc199/volumes"
Oct 30 08:07:49.688496 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:49.688451    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8d20d6-58fd-4888-b638-135de8805efb" path="/var/lib/kubelet/pods/5f8d20d6-58fd-4888-b638-135de8805efb/volumes"
Oct 30 08:07:50.432484 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:50.432446    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:51.063982 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.063938    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:51.182068 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.182026    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:51.206891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.206856    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"]
Oct 30 08:07:51.214925 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.214888    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2eda2c-h76qw"]
Oct 30 08:07:51.688500 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.688466    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f02c5da-2605-4780-8a1a-9499a5a19404" path="/var/lib/kubelet/pods/6f02c5da-2605-4780-8a1a-9499a5a19404/volumes"
Oct 30 08:07:51.747562 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:51.747525    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:57.928371 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:57.928332    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:58.155575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:58.155539    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:07:58.307330 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:58.307223    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 08:07:58.307563 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:58.307538    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy" containerID="cri-o://f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151" gracePeriod=30
Oct 30 08:07:58.337611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:07:58.337579    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:03.398125 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.398087    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:03.567962 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.567929    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 08:08:03.583032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.582939    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.582997    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583028    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583365 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583155    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583365 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583230    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583365 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583345    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583368    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:03.583556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583383    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583426    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583443    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:08:03.583556 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583531    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\") pod \"fe722c8b-006c-476d-a021-1d1df9faa21b\" (UID: \"fe722c8b-006c-476d-a021-1d1df9faa21b\") "
Oct 30 08:08:03.583864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583608    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:03.583864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583623    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:03.583864 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583685    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:03.584012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583897    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.584012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583920    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.584012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583938    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe722c8b-006c-476d-a021-1d1df9faa21b-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.584012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583955    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.584012 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.583968    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.585699 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.585668    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:08:03.586032 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.586005    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:08:03.586138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.586032    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "kube-api-access-w77xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:08:03.586138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.586059    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy" pod "fe722c8b-006c-476d-a021-1d1df9faa21b" (UID: "fe722c8b-006c-476d-a021-1d1df9faa21b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:03.634648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.634588    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:03.684775 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.684738    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.684775 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.684768    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77xg\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-kube-api-access-w77xg\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.684775 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.684778    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.685041 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.684787    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe722c8b-006c-476d-a021-1d1df9faa21b-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:03.804493 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.804460    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:08:03.839833 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.839735    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:03.971893 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.971849    6738 generic.go:356] "Generic (PLEG): container finished" podID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerID="f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151" exitCode=0
Oct 30 08:08:03.972140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.971906    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"
Oct 30 08:08:03.972140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.971926    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" event={"ID":"fe722c8b-006c-476d-a021-1d1df9faa21b","Type":"ContainerDied","Data":"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"}
Oct 30 08:08:03.972140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.972011    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n" event={"ID":"fe722c8b-006c-476d-a021-1d1df9faa21b","Type":"ContainerDied","Data":"8311f2124b71141092bbb7d6ee49844380326587ca28fca759a7e37272e5b306"}
Oct 30 08:08:03.972140 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.972039    6738 scope.go:122] "RemoveContainer" containerID="f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"
Oct 30 08:08:03.985174 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.985148    6738 scope.go:122] "RemoveContainer" containerID="f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"
Oct 30 08:08:03.985558 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:08:03.985534    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151\": container with ID starting with f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151 not found: ID does not exist" containerID="f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"
Oct 30 08:08:03.985692 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:03.985564    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151"} err="failed to get container status \"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151\": rpc error: code = NotFound desc = could not find container \"f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151\": container with ID starting with f8dcc9f8a58853edd48feb285a0137ff4cc300c4e6bfd64639f34aacd07f8151 not found: ID does not exist"
Oct 30 08:08:04.028399 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:04.028361    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 08:08:04.031103 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:04.031044    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:04.033376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:04.033348    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-jk95n"]
Oct 30 08:08:04.714441 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:04.714399    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:08:05.350694 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:05.350662    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:05.402519 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:05.402484    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:08:05.689388 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:05.689354    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" path="/var/lib/kubelet/pods/fe722c8b-006c-476d-a021-1d1df9faa21b/volumes"
Oct 30 08:08:07.517970 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:08:07.517931    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:08:11.122328 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:11.122278    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:17.326423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:17.326384    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:08:18.249589 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:18.249547    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:08:18.886331 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:18.886275    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:08:20.055238 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:20.055203    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2a33320c-xlgm7_20d0c3f2-1adf-47d2-8f47-945ce0939972/revoke-keys/0.log"
Oct 30 08:08:20.260119 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:20.260081    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2cc43303-6kvz2_83bc6742-bffe-482d-a92a-1a934f39276e/revoke-keys/0.log"
Oct 30 08:08:20.451874 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:20.451845    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2d81d4d0-ffpj5_70f5ff51-8a58-4fe3-b7c6-2c956c849938/revoke-keys/0.log"
Oct 30 08:08:20.643543 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:20.643511    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-359c2c46-g6kzn_87624c52-50ff-4dea-a691-f41014ccbd37/revoke-keys/0.log"
Oct 30 08:08:20.841968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:20.841896    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3f424632-4xtm2_8bcdcee7-3dea-471a-bcda-106a5309a382/revoke-keys/0.log"
Oct 30 08:08:21.025882 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:21.025849    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4d5565d0-7nxtl_d6361e7e-4c50-47ef-8ca9-6ca85adce8fb/revoke-keys/0.log"
Oct 30 08:08:21.219908 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:21.219877    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5541922c-kmm5h_28382c65-7d57-4e0e-a0e8-91a3f48a86c3/revoke-keys/0.log"
Oct 30 08:08:21.408190 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:21.408154    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8b61c610-8cp2x_f5c459a9-32b5-4947-8b93-69685cd2c7ec/revoke-keys/0.log"
Oct 30 08:08:21.590535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:21.590447    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8ed4f2cc-9lrhf_eea3646c-555a-43fd-83e8-723ec893c8e4/revoke-keys/0.log"
Oct 30 08:08:21.982713 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:21.982684    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab3ca735-jlvcz_ffab17de-3d1b-40cb-b795-48fa6c546c5d/revoke-keys/0.log"
Oct 30 08:08:22.171534 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:22.171502    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad14e187-2xxpk_e685b84c-5da3-4d34-8d40-446bb69450c2/revoke-keys/0.log"
Oct 30 08:08:22.358082 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:22.358009    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f1577dca-hhv9q_ebddd9f5-5154-4724-a1dc-db09ab270fcf/revoke-keys/0.log"
Oct 30 08:08:28.135857 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:28.135820    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lpfqr_88b5dd99-6942-4130-b01b-00413a307288/discovery/0.log"
Oct 30 08:08:28.365837 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:28.365804    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54599bd898-4gdnn_3039c128-a834-422d-8dbe-5b1bda441bc6/kube-auth-proxy/0.log"
Oct 30 08:08:28.976575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:28.976544    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-7f465_384cf69e-4f91-4b58-8001-875083896c42/istio-proxy/0.log"
Oct 30 08:08:29.179048 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:29.179011    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-hrjtp_22c79cfa-d9a5-4927-a969-42a83e34af9d/istio-proxy/0.log"
Oct 30 08:08:29.379777 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:29.379750    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-skjxh_cb30c606-5ce5-4aa6-9225-cbd6fe5af60c/istio-proxy/0.log"
Oct 30 08:08:31.924514 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:31.924481    6738 scope.go:122] "RemoveContainer" containerID="2540d1986043989c12e8b32d68b786d9ebb14668102942cff4a12a4a8067cd65"
Oct 30 08:08:31.936897 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:31.936868    6738 scope.go:122] "RemoveContainer" containerID="34830ba31475a8a344984a9946e465bde7ef0faeba53e9078132e56b8de67359"
Oct 30 08:08:31.946235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:31.946210    6738 scope.go:122] "RemoveContainer" containerID="aaf3b90b840797595ca36db33e99b38a3aca95b00f93c2a30a78c8a234d08dea"
Oct 30 08:08:31.954823 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:31.954800    6738 scope.go:122] "RemoveContainer" containerID="36a6e4a18d29f0a024edb3e64c3c52e7f135954fdfc3c2f8a5d15e029aa1f3d8"
Oct 30 08:08:31.963676 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:31.963654    6738 scope.go:122] "RemoveContainer" containerID="74ecc93acfcc886a66a3b3d045b784cd562af70eeb5dfd24af343b2bb88fc118"
Oct 30 08:08:32.113235 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:32.113201    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-95n5t_a3fb61a3-1a95-4d09-8c10-56fdcafa1c1f/main/0.log"
Oct 30 08:08:32.745138 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:32.745105    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kwpvn_26c75063-a26f-4073-9561-596ad84493e8/main/0.log"
Oct 30 08:08:32.954946 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:32.954912    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tsl2d_2dc94d72-1c5c-414e-b2e5-189467c1edab/main/0.log"
Oct 30 08:08:36.727006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:36.726970    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:37.516858 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:08:37.516822    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:08:41.575030 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:41.574981    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:41.696211 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:41.696173    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:43.797250 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.797212    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:43.844324 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844273    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pd2d5/must-gather-thmct"]
Oct 30 08:08:43.844672 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844659    6738 cpu_manager.go:414] "containerMap: removing container" podUID="e685b84c-5da3-4d34-8d40-446bb69450c2" containerName="revoke-keys"
Oct 30 08:08:43.844720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844673    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e685b84c-5da3-4d34-8d40-446bb69450c2" containerName="revoke-keys"
Oct 30 08:08:43.844720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844693    6738 cpu_manager.go:414] "containerMap: removing container" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy"
Oct 30 08:08:43.844720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844698    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy"
Oct 30 08:08:43.844720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844715    6738 cpu_manager.go:414] "containerMap: removing container" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main"
Oct 30 08:08:43.844720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844720    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844741    6738 cpu_manager.go:414] "containerMap: removing container" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844746    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844811    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="e685b84c-5da3-4d34-8d40-446bb69450c2" containerName="revoke-keys"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844821    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe722c8b-006c-476d-a021-1d1df9faa21b" containerName="istio-proxy"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844829    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f41d75b-e6a7-443e-8608-d1fb7e161e20" containerName="main"
Oct 30 08:08:43.844917 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.844836    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb29921-bfb1-4e15-b66f-ebb3c6dd05fb" containerName="main"
Oct 30 08:08:43.847647 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.847628    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:43.876000 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.875966    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pd2d5/must-gather-thmct"]
Oct 30 08:08:43.886193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.886157    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pd2d5\"/\"kube-root-ca.crt\""
Oct 30 08:08:43.886193 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.886157    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pd2d5\"/\"openshift-service-ca.crt\""
Oct 30 08:08:43.886434 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.886200    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pd2d5\"/\"default-dockercfg-9tx25\""
Oct 30 08:08:43.974602 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.974561    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fab37d09-d202-42d5-8233-b3f798acaf91-must-gather-output\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:43.974812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.974630    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpcr\" (UniqueName: \"kubernetes.io/projected/fab37d09-d202-42d5-8233-b3f798acaf91-kube-api-access-llpcr\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:43.977568 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:43.977538    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:08:44.075723 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.075627    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fab37d09-d202-42d5-8233-b3f798acaf91-must-gather-output\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:44.075920 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.075727    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpcr\" (UniqueName: \"kubernetes.io/projected/fab37d09-d202-42d5-8233-b3f798acaf91-kube-api-access-llpcr\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:44.075981 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.075960    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fab37d09-d202-42d5-8233-b3f798acaf91-must-gather-output\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:44.100141 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.100107    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpcr\" (UniqueName: \"kubernetes.io/projected/fab37d09-d202-42d5-8233-b3f798acaf91-kube-api-access-llpcr\") pod \"must-gather-thmct\" (UID: \"fab37d09-d202-42d5-8233-b3f798acaf91\") " pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:44.160958 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.160918    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pd2d5/must-gather-thmct"
Oct 30 08:08:44.539575 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.539543    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pd2d5/must-gather-thmct"]
Oct 30 08:08:44.541471 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:08:44.541442    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab37d09_d202_42d5_8233_b3f798acaf91.slice/crio-f1c1e4888fa0fe3cb11e33c28ead6afebe44a9f92f5d0db80429f989311101c4 WatchSource:0}: Error finding container f1c1e4888fa0fe3cb11e33c28ead6afebe44a9f92f5d0db80429f989311101c4: Status 404 returned error can't find the container with id f1c1e4888fa0fe3cb11e33c28ead6afebe44a9f92f5d0db80429f989311101c4
Oct 30 08:08:44.543376 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:44.543358    6738 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 08:08:45.215924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:45.215887    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pd2d5/must-gather-thmct" event={"ID":"fab37d09-d202-42d5-8233-b3f798acaf91","Type":"ContainerStarted","Data":"f1c1e4888fa0fe3cb11e33c28ead6afebe44a9f92f5d0db80429f989311101c4"}
Oct 30 08:08:48.585105 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:48.585062    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2pl2_e2ab3bbf-b449-4b3c-b048-5180eb3b2e37/global-pull-secret-syncer/0.log"
Oct 30 08:08:48.834506 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:48.834472    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 08:08:48.834866 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:48.834696    6738 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy" containerID="cri-o://e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5" gracePeriod=30
Oct 30 08:08:49.549994 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:49.549954    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log"
Oct 30 08:08:50.257182 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:50.257139    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pd2d5/must-gather-thmct" event={"ID":"fab37d09-d202-42d5-8233-b3f798acaf91","Type":"ContainerStarted","Data":"e91f287ce28210f7c03dde46d5905566eaa7a46413d7e5ce751491c1eac95cc8"}
Oct 30 08:08:50.257740 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:50.257195    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pd2d5/must-gather-thmct" event={"ID":"fab37d09-d202-42d5-8233-b3f798acaf91","Type":"ContainerStarted","Data":"95d283a0da4ea98ba44b73ba32912a1384c0495905900c31ebd42562aaf5f53d"}
Oct 30 08:08:50.261622 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:50.261583    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-124.ec2.internal_bc1bcd324f5af093ecce999f61f1d18f/haproxy/0.log"
Oct 30 08:08:50.299440 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:50.299376    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pd2d5/must-gather-thmct" podStartSLOduration=2.289964052 podStartE2EDuration="7.299354504s" totalImagesPullingTime="5.009390452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:08:50.298267312 +0000 UTC m=+4290.731031205" watchObservedRunningTime="2026-10-30 08:08:50.299354504 +0000 UTC m=+4290.732118395"
Oct 30 08:08:51.926423 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:51.926384    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:08:51.979726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:51.979686    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:08:52.036072 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:52.036027    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"]
Oct 30 08:08:52.048880 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:52.048826    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"]
Oct 30 08:08:52.050642 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:52.050612    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3ca735-jlvcz"]
Oct 30 08:08:52.053099 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:52.053072    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5565d0-7nxtl"]
Oct 30 08:08:53.689001 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:53.688914    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6361e7e-4c50-47ef-8ca9-6ca85adce8fb" path="/var/lib/kubelet/pods/d6361e7e-4c50-47ef-8ca9-6ca85adce8fb/volumes"
Oct 30 08:08:53.689627 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:53.689569    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffab17de-3d1b-40cb-b795-48fa6c546c5d" path="/var/lib/kubelet/pods/ffab17de-3d1b-40cb-b795-48fa6c546c5d/volumes"
Oct 30 08:08:53.861504 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:53.861460    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:08:54.011483 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.011450    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:08:54.112202 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.112172    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 08:08:54.184256 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184222    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.184572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184269    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.184572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184328    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.184572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184366    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.184572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184501    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.184572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184573    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.185112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184660    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.185112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184704    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.185112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184795    6738 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\") pod \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\" (UID: \"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c\") "
Oct 30 08:08:54.185112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184920    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:08:54.185112 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.184914    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:54.185480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.185208    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:54.185480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.185267    6738 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istiod-ca-cert\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.185480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.185288    6738 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-data\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.185480 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.185389    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:54.185724 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.185548    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:54.187580 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.187068    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "kube-api-access-657gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:08:54.187580 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.187213    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:08:54.187580 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.187434    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:08:54.187891 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.187840    6738 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo" pod "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" (UID: "cb30c606-5ce5-4aa6-9225-cbd6fe5af60c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:08:54.284118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.284031    6738 generic.go:356] "Generic (PLEG): container finished" podID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerID="e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5" exitCode=0
Oct 30 08:08:54.284118 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.284077    6738 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"
Oct 30 08:08:54.284349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.284126    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" event={"ID":"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c","Type":"ContainerDied","Data":"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"}
Oct 30 08:08:54.284349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.284209    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh" event={"ID":"cb30c606-5ce5-4aa6-9225-cbd6fe5af60c","Type":"ContainerDied","Data":"afe3f054388ddc43bdb01d4b5112d3ef1eff041e52a3906af76e6f26f8d9c381"}
Oct 30 08:08:54.284349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.284236    6738 scope.go:122] "RemoveContainer" containerID="e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"
Oct 30 08:08:54.285749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285726    6738 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285749 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285750    6738 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-workload-certs\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285760    6738 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-credential-socket\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285769    6738 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-podinfo\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285777    6738 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657gk\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-kube-api-access-657gk\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285787    6738 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-envoy\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.285909 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.285795    6738 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c-istio-token\") on node \"ip-10-0-139-124.ec2.internal\" DevicePath \"\""
Oct 30 08:08:54.295895 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.295877    6738 scope.go:122] "RemoveContainer" containerID="e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"
Oct 30 08:08:54.296254 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:08:54.296228    6738 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5\": container with ID starting with e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5 not found: ID does not exist" containerID="e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"
Oct 30 08:08:54.296365 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.296265    6738 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5"} err="failed to get container status \"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5\": rpc error: code = NotFound desc = could not find container \"e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5\": container with ID starting with e01006015c6ff3046957516d891c052c0790bfee7ca1f7b8bc707df0ee4ad2a5 not found: ID does not exist"
Oct 30 08:08:54.342907 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.342869    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 08:08:54.345581 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:54.345544    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skjxh"]
Oct 30 08:08:55.689535 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:55.689503    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" path="/var/lib/kubelet/pods/cb30c606-5ce5-4aa6-9225-cbd6fe5af60c/volumes"
Oct 30 08:08:57.807886 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:57.807851    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:08:59.047969 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.047935    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:08:59.354982 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.354898    6738 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"]
Oct 30 08:08:59.355318 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.355286    6738 cpu_manager.go:414] "containerMap: removing container" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy"
Oct 30 08:08:59.355318 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.355312    6738 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy"
Oct 30 08:08:59.355429 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.355408    6738 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb30c606-5ce5-4aa6-9225-cbd6fe5af60c" containerName="istio-proxy"
Oct 30 08:08:59.357726 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.357702    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.377111 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.377068    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"]
Oct 30 08:08:59.439392 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.439357    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-sys\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.439598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.439481    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-lib-modules\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.439598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.439520    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4xr\" (UniqueName: \"kubernetes.io/projected/e0478269-c411-4e4c-9869-ecb743e6dfb0-kube-api-access-6c4xr\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.439598 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.439538    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-podres\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.439709 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.439692    6738 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-proc\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.465006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.464971    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tpf\"/\"openshift-service-ca.crt\""
Oct 30 08:08:59.465006 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.464982    6738 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h4tpf\"/\"default-dockercfg-x222n\""
Oct 30 08:08:59.465271 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.465057    6738 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tpf\"/\"kube-root-ca.crt\""
Oct 30 08:08:59.541286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541240    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-proc\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541286 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541304    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-sys\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541381    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-proc\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541388    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-lib-modules\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541444    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-sys\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541467    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4xr\" (UniqueName: \"kubernetes.io/projected/e0478269-c411-4e4c-9869-ecb743e6dfb0-kube-api-access-6c4xr\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541491    6738 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-podres\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541576 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541454    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-lib-modules\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.541816 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.541598    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0478269-c411-4e4c-9869-ecb743e6dfb0-podres\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.565350 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.565318    6738 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4xr\" (UniqueName: \"kubernetes.io/projected/e0478269-c411-4e4c-9869-ecb743e6dfb0-kube-api-access-6c4xr\") pod \"perf-node-gather-daemonset-5tnth\" (UID: \"e0478269-c411-4e4c-9869-ecb743e6dfb0\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.669945 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.669911    6738 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:08:59.984648 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:08:59.984618    6738 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"]
Oct 30 08:08:59.986772 ip-10-0-139-124 kubenswrapper[6738]: W1030 08:08:59.986738    6738 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0478269_c411_4e4c_9869_ecb743e6dfb0.slice/crio-398ba8adb2e129d367ebc41effd1905cbee47d63b585ab6a6392bb5cf119a8e8 WatchSource:0}: Error finding container 398ba8adb2e129d367ebc41effd1905cbee47d63b585ab6a6392bb5cf119a8e8: Status 404 returned error can't find the container with id 398ba8adb2e129d367ebc41effd1905cbee47d63b585ab6a6392bb5cf119a8e8
Oct 30 08:09:00.293114 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:00.293035    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:09:00.323539 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:00.323500    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth" event={"ID":"e0478269-c411-4e4c-9869-ecb743e6dfb0","Type":"ContainerStarted","Data":"0417cd46a8ee0515a4678f616216ad3c0a57f716c290dfbdfe8e40dd5210fcaa"}
Oct 30 08:09:00.323770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:00.323564    6738 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth" event={"ID":"e0478269-c411-4e4c-9869-ecb743e6dfb0","Type":"ContainerStarted","Data":"398ba8adb2e129d367ebc41effd1905cbee47d63b585ab6a6392bb5cf119a8e8"}
Oct 30 08:09:00.323770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:00.323607    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:09:00.372924 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:00.372867    6738 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth" podStartSLOduration=1.37285222 podStartE2EDuration="1.37285222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:00.371497487 +0000 UTC m=+4300.804261380" watchObservedRunningTime="2026-10-30 08:09:00.37285222 +0000 UTC m=+4300.805616112"
Oct 30 08:09:03.422259 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:03.422220    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:09:03.968056 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:03.968021    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:09:04.206720 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:04.206685    6738 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djbct_7c36d37c-2e1a-41aa-9f1a-bf67e50331c9/konnectivity-agent/0.log" limit=1024
Oct 30 08:09:06.343770 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:06.343697    6738 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-5tnth"
Oct 30 08:09:06.521627 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:06.521591    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:09:07.520274 ip-10-0-139-124 kubenswrapper[6738]: E1030 08:09:07.520228    6738 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rqlss" containerName="extractor"
Oct 30 08:09:09.107689 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:09.107653    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/alertmanager/0.log"
Oct 30 08:09:09.865681 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:09.865650    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/config-reloader/0.log"
Oct 30 08:09:10.129314 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:10.129260    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:09:10.749968 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:10.749942    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/kube-rbac-proxy-web/0.log"
Oct 30 08:09:11.383327 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:11.383286    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/kube-rbac-proxy/0.log"
Oct 30 08:09:11.512540 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:11.512505    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/kube-rbac-proxy-metric/0.log"
Oct 30 08:09:11.651650 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:11.651561    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/prom-label-proxy/0.log"
Oct 30 08:09:11.792279 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:11.792247    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aab7e92d-aa0e-4568-9c8f-7d76b5968cfd/init-config-reloader/0.log"
Oct 30 08:09:11.953349 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:11.953224    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6chbj_afbea8a0-a4f1-489c-97f0-25b42a9e9239/proxy/0.log"
Oct 30 08:09:12.079255 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:12.079219    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8khnk_0f2a8f9c-d148-4051-855b-79b8da1b57dc/kube-state-metrics/0.log"
Oct 30 08:09:12.204245 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:12.204140    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8khnk_0f2a8f9c-d148-4051-855b-79b8da1b57dc/kube-rbac-proxy-main/0.log"
Oct 30 08:09:12.204551 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:12.204527    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:09:12.567017 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:12.566940    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8khnk_0f2a8f9c-d148-4051-855b-79b8da1b57dc/kube-rbac-proxy-self/0.log"
Oct 30 08:09:12.866611 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:12.866520    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-n75wj_17f2a963-5150-4193-826b-bfd5a336ce5b/monitoring-plugin/0.log"
Oct 30 08:09:14.030741 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:14.030705    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"]
Oct 30 08:09:14.043530 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:14.043492    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b61c610-8cp2x"]
Oct 30 08:09:14.168609 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:14.168567    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4kpg_cff3c19d-9829-428c-a1c0-a5b623913630/manager/0.log"
Oct 30 08:09:14.396812 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:14.396772    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7df767b8fd-g9dgt_b0c9cc41-c7e2-498c-812f-93268565f166/manager/0.log"
Oct 30 08:09:15.076641 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:15.076419    6738 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"]
Oct 30 08:09:15.087370 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:15.087339    6738 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1577dca-hhv9q"]
Oct 30 08:09:15.649758 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:15.649721    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vns7r_f6830ae7-bf50-490b-bbf8-f5d32835edfb/node-exporter/0.log"
Oct 30 08:09:15.690357 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:15.690318    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebddd9f5-5154-4724-a1dc-db09ab270fcf" path="/var/lib/kubelet/pods/ebddd9f5-5154-4724-a1dc-db09ab270fcf/volumes"
Oct 30 08:09:15.690923 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:15.690897    6738 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c459a9-32b5-4947-8b93-69685cd2c7ec" path="/var/lib/kubelet/pods/f5c459a9-32b5-4947-8b93-69685cd2c7ec/volumes"
Oct 30 08:09:16.462845 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:16.462811    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vns7r_f6830ae7-bf50-490b-bbf8-f5d32835edfb/kube-rbac-proxy/0.log"
Oct 30 08:09:18.007572 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:18.007538    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vns7r_f6830ae7-bf50-490b-bbf8-f5d32835edfb/init-textfile/0.log"
Oct 30 08:09:19.108896 ip-10-0-139-124 kubenswrapper[6738]: I1030 08:09:19.108856    6738 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lpfqr_88b5dd99-6942-4130-b01b-00413a307288/discovery/0.log"